__ __ / \ / \ ______ _ _ __ __ __ / /\ / /\ \ / __ \ | | | | / \ \ \ / / / / / / \ \ | | | | | | | | / /\ \ \ / \ \ \ \ / / | |__| | | |__| | / ____ \ | | \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| \__/ \__/ \ \__ \___\ by Red Hat Build, Store, and Distribute your Containers Startup timestamp: Tue Jan 17 04:49:40 UTC 2023 Running all default registry services without migration Running init script '/quay-registry/conf/init/certs_install.sh' Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory Running init script '/quay-registry/conf/init/copy_config_files.sh' Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh' Validating Configuration time="2023-01-17T04:49:40Z" level=debug msg="Validating AccessSettings" time="2023-01-17T04:49:40Z" level=debug msg="Validating ActionLogArchiving" time="2023-01-17T04:49:40Z" level=debug msg="Validating AppTokenAuthentication" time="2023-01-17T04:49:40Z" level=debug msg="Validating BitbucketBuildTrigger" time="2023-01-17T04:49:40Z" level=debug msg="bitbucket response: 400, {\"error\": \"invalid_grant\", \"error_description\": \"The specified code is not valid.\"}" time="2023-01-17T04:49:40Z" level=debug msg="Validating BuildManager" time="2023-01-17T04:49:40Z" level=debug msg="Validating Database" time="2023-01-17T04:49:40Z" level=debug msg="Scheme: postgresql" time="2023-01-17T04:49:40Z" level=debug msg="Host: quayregistry-quay-database:5432" time="2023-01-17T04:49:40Z" level=debug msg="Db: quayregistry-quay-database" time="2023-01-17T04:49:40Z" level=debug msg="Params: " time="2023-01-17T04:49:40Z" level=debug msg="Including params " time="2023-01-17T04:49:40Z" level=debug msg="Pinging database at postgresql://quayregistry-quay-database:A0wE7b-Mv8cs1PgSnHaCuX3YAbtMKF1Aodz4cvKHu5mmCSqpoTsvUnE8iRfsQoi6WlAQc6u7ZlMbl8dL@quayregistry-quay-database:5432/quayregistry-quay-database" plpgsql pg_trgm time="2023-01-17T04:49:40Z" level=debug msg="Validating DistributedStorage" time="2023-01-17T04:49:41Z" level=debug msg="Validating ElasticSearch" time="2023-01-17T04:49:41Z" level=debug msg="Validating Email" time="2023-01-17T04:49:41Z" level=debug msg="Validating GitHubBuildTrigger" time="2023-01-17T04:49:41Z" level=debug msg="github response: 200, \n\n\n\n\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n GitHub: Let’s build from here · GitHub\n\n\n\n \n\n \n\n\n\n \n\n \n \n\n \n\n \n \n \n \n \n\n\n\n \n\n \n\n\n\n\n \n\n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n\n\n \n\n\n\n \n\n \n\n\n \n \n \n \n\n \n\n \n \n \n\n \n\n \n \n \n\n\n\n \n \n\n\n \n\n \n\n \n\n \n \n \n\n\n\n\n\n \n\n \n\n \n
\n \n\n\n
\n Skip to content\n \n \n \n \n\n\n \n\n \n
\n \n\n
\n
\n \n \n \n\n \n\n
\n \n Sign up\n \n
\n\n
\n \n
\n
\n\n\n
\n
\n \n\n
\n
\n \n\n\n\n
\n
\n
\n \n
\n
\n\n
\n\n
\n \n Sign in\n \n
\n\n \n Sign up\n \n
\n
\n
\n
\n
\n\n
\n\n
\n\n\n\n\n\n\n\n
\n\n\n\n\n\n \n
\n\n\n \n \n\n\n\n\n\n\n \n
\n\n\n
\n
\n \n \n \n\n \"\"\n \n
\n\n
\n
\n \n\n
\n
\n \"\"\n\n
\n
\n \n \n\n \n
\n
\n\n
\n
\n\n
\n
\n \n
\n\n

\n Let’s build from here\n

\n\n

\n Harnessed for productivity. Designed for collaboration. Celebrated for built-in security. Welcome to the platform developers love.\n

\n\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n \n
\n
\n
\n\n \n Start a free enterprise trial\n \n \n\n\n
\n\n
\n

Trusted by the world’s leading organizations ↘︎

\n\n
\n \"Stripe\n \"Pinterest\n
\n \"KPMG\n
\n \"Mercedes-Benz\n
\n \"P&G\n \"Telus\n
\n
\n
\n
\n
\n
\n\n
\n
\n
\n \n\n \n\n
\n Productivity\n Collaboration\n Security\n
\n\n
\n \n Start a free enterprise trial\n \n \n\n\n \n Sign up for GitHub\n \n \n\n\n
\n
\n
\n
\n\n
\n
\n \n
\n
\n
\n \n \n\n \n
\n\n
\n
\n\n
\n

Productivity

\n

\n Accelerate high-quality software development.\n Our platform drives innovation with tools that boost developer velocity.\n

\n
\n
\n\n
\n
\n
\n \n \n\n \n \n\n
\n
\n \n \n\n
mona-github-github-g59jpq2w5w7.github.dev
\n \n \n\n
\n
\n \n \n\n
\n
\n\n
\n
\n \n \n\n
\n\n
\n
\n
    \n
  • index.html
    • \n
  • script.js
    • \n
  • package.json
    • \n
\n\n
\n
\n
\n
1
\n
2
\n
3
\n
4
\n
5
\n
6
\n
7
\n
8
\n
9
\n
10
\n
11
\n
12
\n
\n\n
\n 
<div class="position-absolute width-full color-bg-default" style="bottom: -4rem;">
\n 
  <div class="container-xl p-responsive">
\n 
    <div class="d-flex flex-justify-center flex-lg-justify-end color-bg-default">
\n 
      <div class="col-8 col-sm-7 col-md-6 col-lg-5 position-relative z-2 right-lg-n12 events-none">
\n 
        <picture>
\n 
          <source srcset="astro-mona.webp" type="image/webp">
\n 
          <img src="astro-mona.svg" width="960" height="967" class="home-astro-mona width-full position-absolute bottom-0 height-auto" alt="Mona looking at GitHub activity across the globe">
\n 
        </picture>
\n 
      </div>
\n 
    </div>
\n 
  </div>
\n 
</div>
\n\n
\n
\n\n\n
\n
\n\n
\n
    \n
  • Terminal
    • \n
  • Output
    • \n
  • Problems
    • \n
  • Debug Console
    • \n
\n\n
\n
\n [09:43:36]\n Starting\n 'watch-extension:vscode-api-tests'\n ...\n
\n
\n [09:43:36]\n Finished\n 'clean-extension:typescript-language-features'\n after\n 248 ms\n
\n
\n [09:43:36]\n Starting\n 'watch-extension:typescript-language-features'\n ...\n
\n
\n [09:43:36]\n Finished\n 'clean-extension:php-language-features'\n after\n 384 ms\n
\n
\n [09:43:36]\n Starting\n 'watch-extension:php-language-features'\n ...\n
\n
\n [09:43:40]\n Finished\n 'clean-extension:html-language-features-server'\n after\n 4.66 s\n
\n
\n [09:43:40]\n Starting\n 'watch-extension:html-language-features-server'\n ...\n
\n
\n [09:43:43]\n Finished\n 'clean-client'\n after\n 7.33 s\n
\n
\n [09:43:43]\n Starting\n 'watch-client'\n ...\n
\n
\n
\n
\n\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n\n
\n \n \n \n \"Planet\n\n
\n
\n
\n
\n\n\n
\n \"\"\n\n
\n
\n \"\"\n
\n\n
\n
\n
\n
\n \n \n \n \n

Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere.

\n \n \n Check out Codespaces \n\n\n \n
\n
\n\n
\n
\n \n
\n
\n \"\"\n \"\"\n
\n
\n \"\"\n
\n
\n\n
\n
\n
\n
\n
\n
\n Did you know?\n
\n\n

22% increase

\n

in developer productivity
after three years with GitHub1

\n
\n
\n
\n
\n\n
\n
\n

GitHub Copilot is your AI pair programmer that empowers you to complete tasks 55% faster by turning natural language prompts into coding suggestions.

\n\n \n
\n \n
\n
\n \n\n \n
\n \n
\n\n
\n
\n
\n
1
\n
2
\n
3
\n
4
\n
5
\n
6
\n
7
\n
8
\n
\n\n
\n 
import matplotlib.pyplot as plt
\n
\n 
def draw_scatterplot(x_values, y_values):
\n 
    plt.scatter(x_values, y_values, s=20)
\n 
    plt.title("Scatter Plot")
\n 
    plt.xlabel("x values")
\n 
    plt.ylabel("y values")
\n 
    plt.show()
\n\n
\n \n\n Copilot\n
\n
\n
\n\n
\n
\n\n
\n
\n
\n
\n
\n \n
\n
\n
\n

GitHub Actions automates your build, test, and deployment workflow with simple and secure CI/CD.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n
\n
\n

GitHub Mobile fits your projects in your pocket, so you never miss a beat while on the go.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n\n
\n
\n
\n\n
\n
\n
\n
\n
\n \n
\n
\n
\n \n \n\n \n
\n\n
\n
\n\n
\n

Collaboration

\n

\n Supercharge collaboration.\n We provide unlimited repositories, best-in-class version control, and the world’s most powerful open source community—so your team can work more efficiently together.\n

\n
\n
\n\n \n \n \n \"Illustration\n\n\n
\n \"\"\n\n
\n
\n \"\"\n
\n\n
\n
\n
\n
\n \n \n \n \n

GitHub Issues and GitHub Projects supply flexible project management tools that adapt to your team alongside your code.

\n \n \n Explore GitHub Issues \n\n\n \n
\n
\n\n
\n
\n \n
\n
\n \"\"\n
\n
\n\n
\n
\n
\n
\n
\n
\n Did you know?\n
\n\n

80% reduction

\n

in onboarding time with GitHub1

\n
\n
\n
\n
\n\n
\n \n
\n
\n
\n

GitHub Discussions create space to ask questions and have open-ended conversations.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n
\n
\n

Pull requests allow real-time communication and collaboration about code changes.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n\n
\n
\n

GitHub Sponsors lets you support your favorite open source maintainers and projects.

\n\n
\n Invest with GitHub Sponsors \n\n
\n
\n \n
\n
\n
\n \n \"Nikema\n
Nikema
\n \n
\n
\n \n \"Directus\n
Directus
\n \n
\n
\n \n \"kazuya\n
kazuya kawaguchi
\n \n
\n
\n \n \"sindresorhus\n
sindresorhus
\n \n
\n
\n \n \"curl\n
curl
\n \n
\n
\n \n \"Homebrew\n
Homebrew
\n \n
\n
\n \n \"ESLint\n
ESLint
\n \n
\n
\n \n \"Evan\n
Evan You
\n \n
\n
\n \n \"CommandPost\n
CommandPost
\n \n
\n
\n \n \"Nick\n
Nick DeJesus
\n \n
\n
\n \n \"chaynHQ\n
chaynHQ
\n \n
\n
\n \n \"Samuel\n
Samuel
\n \n
\n
\n
\n
\n
\n
\n
\n\n
\n
\n
\n
\n \n
\n
\n
\n \n \n\n \n
\n\n
\n
\n\n
\n

Security

\n

\n Embed security into the developer workflow.\n With GitHub, developers can secure their code in minutes and organizations can automatically comply with regulations.\n

\n
\n
\n\n
\n
\n
cmake.yml
\n on: push\n
\n\n
\n
\n
\n
    \n
  • \n
    \n \n \n\n Build\n
    \n 1m 21s\n
    • \n
\n
\n
\n\n
\n
\n
\n
\n
\n\n
\n
\n Steps\n
\n
\n
    \n
  • \n
    \n \n \n\n Initialize CodeQL\n
    \n 1m 42s\n
    • \n
  • \n
    \n \n \n\n Autobuild\n
    \n 1m 24s\n
    • \n
  • \n
    \n \n \n\n Perform CodeQL Analyses\n
    \n 1m 36s\n
    • \n
\n
\n
\n
\n
\n\n
\n \"\"\n\n
\n
\n \"\"\n
\n\n
\n
\n
\n
\n \n \n \n \n

GitHub Advanced Security lets you gain visibility into your security posture, respond to threats proactively, and ship secure applications quickly.

\n \n \n Get GitHub Advanced Security \n\n\n \n
\n
\n\n
\n
\n \n
\n
\n \"\"\n
\n
\n\n
\n
\n
\n
\n
\n
\n Did you know?\n
\n\n

56 million projects

\n

fixed vulnerabilities with GitHub2

\n
\n
\n
\n
\n\n
\n
\n
\n

Secret scanning automatically looks for partner patterns and prevents fraudulent use of accidentally committed secrets.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n \n
\n
\n
\n

Dependabot makes it easy to find and fix vulnerable dependencies in your supply chain.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n
\n
\n

Code scanning is GitHub’s static code analysis tool that helps you remediate issues in your code.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n\n
\n
\n\n
\n \n \n
\n
\n
\n
\n \n \n \n \n

The place for anyone from anywhere to build anything

\n

Whether you’re scaling your startup or just learning how to code, GitHub is your home. Join the world’s largest developer platform to build the innovations that empower humanity. Let’s build from here.

\n \n
\n\n
\n \n Sign up for GitHub\n \n \n\n\n \n Start a free enterprise trial\n \n \n\n\n
\n\n
\n\n
\n
\n\n
\n\n
\n
\n
\n \n \n \n\n \n\n
\n
\n\n \n
\n\n\n \n \n \n \"\"\n\n\n
\n \"Mona\n
\n\n \n\n\n \n
\n
\n \n
\n\n\n\n \n\n \n\n\n\n\n \n\n
\n \n \n\n \n \n
\n \n\n
\n
\n
\n
\n\n \n\n\n\n\n \n\n
\n \n\n\n" time="2023-01-17T04:49:41Z" level=debug msg="Validating GitHubLogin" time="2023-01-17T04:49:41Z" level=debug msg="Validating GitLabBuildTrigger" time="2023-01-17T04:49:41Z" level=debug msg="Validating GoogleLogin" time="2023-01-17T04:49:41Z" level=debug msg="Validating HostSettings" time="2023-01-17T04:49:41Z" level=debug msg="Validating JWTAuthentication" time="2023-01-17T04:49:41Z" level=debug msg="Validating LDAP" time="2023-01-17T04:49:41Z" level=debug msg="Validating OIDC" time="2023-01-17T04:49:41Z" level=debug msg="Validating QuayDocumentation" time="2023-01-17T04:49:41Z" level=debug msg="Validating Redis" time="2023-01-17T04:49:41Z" level=debug msg="Address: quayregistry-quay-redis:6379" time="2023-01-17T04:49:41Z" level=debug msg="Username: " time="2023-01-17T04:49:41Z" level=debug msg="Password Len: 0" time="2023-01-17T04:49:41Z" level=debug msg="Ssl: " time="2023-01-17T04:49:41Z" level=debug msg="Address: quayregistry-quay-redis:6379" time="2023-01-17T04:49:41Z" level=debug msg="Username: " time="2023-01-17T04:49:41Z" level=debug msg="Password Len: 0" time="2023-01-17T04:49:41Z" level=debug msg="Ssl: " time="2023-01-17T04:49:41Z" level=debug msg="Validating RepoMirror" time="2023-01-17T04:49:41Z" level=debug msg="Validating SecurityScanner" time="2023-01-17T04:49:41Z" level=debug msg="Validating TeamSyncing" time="2023-01-17T04:49:41Z" level=debug msg="Validating TimeMachine" time="2023-01-17T04:49:41Z" level=debug msg="Validating UserVisibleSettings" +------------------------+-------+--------+ | Field Group | Error | Status | +------------------------+-------+--------+ | AccessSettings | - | 🟢 | +------------------------+-------+--------+ | ActionLogArchiving | - | 🟢 | +------------------------+-------+--------+ | AppTokenAuthentication | - | 🟢 | +------------------------+-------+--------+ | BitbucketBuildTrigger | - | 🟢 | +------------------------+-------+--------+ | BuildManager | - | 🟢 | +------------------------+-------+--------+ | Database | - | 🟢 | +------------------------+-------+--------+ | DistributedStorage | - | 🟢 | +------------------------+-------+--------+ | ElasticSearch | - | 🟢 | +------------------------+-------+--------+ | Email | - | 🟢 | +------------------------+-------+--------+ | GitHubBuildTrigger | - | 🟢 | +------------------------+-------+--------+ | GitHubLogin | - | 🟢 | +------------------------+-------+--------+ | GitLabBuildTrigger | - | 🟢 | +------------------------+-------+--------+ | GoogleLogin | - | 🟢 | +------------------------+-------+--------+ | HostSettings | - | 🟢 | +------------------------+-------+--------+ | JWTAuthentication | - | 🟢 | +------------------------+-------+--------+ | LDAP | - | 🟢 | +------------------------+-------+--------+ | OIDC | - | 🟢 | +------------------------+-------+--------+ | QuayDocumentation | - | 🟢 | +------------------------+-------+--------+ | Redis | - | 🟢 | +------------------------+-------+--------+ | RepoMirror | - | 🟢 | +------------------------+-------+--------+ | SecurityScanner | - | 🟢 | +------------------------+-------+--------+ | TeamSyncing | - | 🟢 | +------------------------+-------+--------+ | TimeMachine | - | 🟢 | +------------------------+-------+--------+ | UserVisibleSettings | - | 🟢 | +------------------------+-------+--------+ Running init script '/quay-registry/conf/init/nginx_conf_create.sh' Running init script '/quay-registry/conf/init/supervisord_conf_create.sh' Running init script '/quay-registry/conf/init/zz_boot.sh' 2023-01-17 04:49:43,150 INFO RPC interface 'supervisor' initialized 2023-01-17 04:49:43,150 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2023-01-17 04:49:43,150 INFO supervisord started with pid 7 2023-01-17 04:49:44,152 INFO spawned: 'stdout' with pid 58 2023-01-17 04:49:44,153 INFO spawned: 'blobuploadcleanupworker' with pid 59 2023-01-17 04:49:44,154 INFO spawned: 'builder' with pid 60 2023-01-17 04:49:44,155 INFO spawned: 'buildlogsarchiver' with pid 61 2023-01-17 04:49:44,158 INFO spawned: 'chunkcleanupworker' with pid 62 2023-01-17 04:49:44,158 INFO spawned: 'dnsmasq' with pid 63 2023-01-17 04:49:44,160 INFO spawned: 'expiredappspecifictokenworker' with pid 64 2023-01-17 04:49:44,161 INFO spawned: 'exportactionlogsworker' with pid 65 2023-01-17 04:49:44,163 INFO spawned: 'gcworker' with pid 66 2023-01-17 04:49:44,164 INFO spawned: 'globalpromstats' with pid 67 2023-01-17 04:49:44,166 INFO spawned: 'gunicorn-registry' with pid 68 2023-01-17 04:49:44,167 INFO spawned: 'gunicorn-secscan' with pid 69 2023-01-17 04:49:44,168 INFO spawned: 'gunicorn-web' with pid 70 2023-01-17 04:49:44,170 INFO spawned: 'logrotateworker' with pid 71 2023-01-17 04:49:44,171 INFO spawned: 'manifestbackfillworker' with pid 72 2023-01-17 04:49:44,173 INFO spawned: 'memcache' with pid 73 2023-01-17 04:49:44,174 INFO spawned: 'namespacegcworker' with pid 74 2023-01-17 04:49:44,226 INFO spawned: 'nginx' with pid 75 2023-01-17 04:49:44,228 INFO spawned: 'notificationworker' with pid 76 2023-01-17 04:49:44,230 INFO spawned: 'pushgateway' with pid 77 2023-01-17 04:49:44,231 INFO spawned: 'queuecleanupworker' with pid 78 2023-01-17 04:49:44,233 INFO spawned: 'repositoryactioncounter' with pid 84 2023-01-17 04:49:44,235 INFO spawned: 'repositorygcworker' with pid 85 2023-01-17 04:49:44,237 INFO spawned: 'securityscanningnotificationworker' with pid 86 2023-01-17 04:49:44,325 INFO spawned: 'securityworker' with pid 87 2023-01-17 04:49:44,328 INFO spawned: 'servicekey' with pid 88 2023-01-17 04:49:44,330 INFO spawned: 'storagereplication' with pid 89 2023-01-17 04:49:44,332 INFO spawned: 'teamsyncworker' with pid 90 2023-01-17 04:49:46,136 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,136 INFO success: blobuploadcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,136 INFO success: builder entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,136 INFO success: buildlogsarchiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,136 INFO success: chunkcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,136 INFO success: dnsmasq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,136 INFO success: expiredappspecifictokenworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: exportactionlogsworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: gcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: globalpromstats entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: gunicorn-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: gunicorn-secscan entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: gunicorn-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: logrotateworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: manifestbackfillworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: memcache entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: namespacegcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: notificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: pushgateway entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: queuecleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: repositoryactioncounter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: repositorygcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: securityscanningnotificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: securityworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: servicekey entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: storagereplication entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:46,137 INFO success: teamsyncworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) dnsmasq stderr | dnsmasq: started, version 2.79 cachesize 150 dnsmasq stderr | dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dnsmasq stderr | dnsmasq: reading /etc/resolv.conf dnsmasq stderr | dnsmasq: using nameserver 172.30.0.10#53 dnsmasq stderr | dnsmasq: read /etc/hosts - 7 addresses nginx stdout | 2023/01/17 04:49:44 [warn] 75#0: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:40 nginx stdout | 2023/01/17 04:49:44 [warn] 75#0: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:67 nginx stdout | 2023/01/17 04:49:44 [warn] 75#0: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:91 nginx stderr | nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:40 nginx stderr | nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:67 nginx stderr | nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:91 nginx stdout | 2023/01/17 04:49:44 [alert] 103#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 100#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 99#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 101#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 113#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 114#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 107#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 108#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 104#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 109#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 110#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 102#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 112#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 106#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:44 [alert] 105#0: setpriority(-10) failed (13: Permission denied) pushgateway stderr | level=info ts=2023-01-17T04:49:44.929Z caller=main.go:85 msg="starting pushgateway" version="(version=, branch=, revision=)" pushgateway stderr | level=info ts=2023-01-17T04:49:44.929Z caller=main.go:86 build_context="(go=go1.17.12, user=, date=)" nginx stdout | 2023/01/17 04:49:44 [alert] 111#0: setpriority(-10) failed (13: Permission denied) pushgateway stderr | level=info ts=2023-01-17T04:49:45.126Z caller=main.go:139 listen_address=:9091 pushgateway stderr | level=info ts=2023-01-17T04:49:45.127Z caller=tls_config.go:191 msg="TLS is disabled." http2=false nginx stdout | 2023/01/17 04:49:52 [crit] 100#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.131.0.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.131.0.22:8443" nginx stdout | 2023/01/17 04:49:52 [crit] 100#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.131.0.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.131.0.22:8443" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /health/instance HTTP/2.0" 502 157 "-" "kube-probe/1.24" (0.000 47 0.000 : 0.000) teamsyncworker stdout | 2023-01-17 04:50:01,431 [90] [DEBUG] [__main__] Team syncing is disabled; sleeping servicekey stdout | 2023-01-17 04:50:01,636 [88] [DEBUG] [workers.worker] Scheduling worker. servicekey stdout | 2023-01-17 04:50:01,726 [88] [INFO] [apscheduler.scheduler] Scheduler started servicekey stdout | 2023-01-17 04:50:02,027 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 04:50:02,028 [88] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added servicekey stdout | 2023-01-17 04:50:02,028 [88] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:00:27.029240 servicekey stdout | 2023-01-17 04:50:02,631 [88] [INFO] [apscheduler.scheduler] Added job "ServiceKeyWorker._refresh_service_key" to job store "default" servicekey stdout | 2023-01-17 04:50:02,632 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 04:50:02,632 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:27.029240+00:00 (in 624.396979 seconds) securityworker stdout | 2023-01-17 04:50:02,636 [87] [DEBUG] [workers.worker] Scheduling worker. securityworker stdout | 2023-01-17 04:50:02,637 [87] [INFO] [apscheduler.scheduler] Scheduler started securityworker stdout | 2023-01-17 04:50:02,825 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:02,825 [87] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added securityworker stdout | 2023-01-17 04:50:02,836 [87] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:22.837365 buildlogsarchiver stdout | 2023-01-17 04:50:03,132 [61] [DEBUG] [workers.worker] Scheduling worker. buildlogsarchiver stdout | 2023-01-17 04:50:03,133 [61] [INFO] [apscheduler.scheduler] Scheduler started buildlogsarchiver stdout | 2023-01-17 04:50:03,135 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:50:03,135 [61] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added buildlogsarchiver stdout | 2023-01-17 04:50:03,225 [61] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:27.226424 manifestbackfillworker stdout | 2023-01-17 04:50:03,533 [72] [DEBUG] [workers.worker] Scheduling worker. manifestbackfillworker stdout | 2023-01-17 04:50:03,534 [72] [INFO] [apscheduler.scheduler] Scheduler started manifestbackfillworker stdout | 2023-01-17 04:50:03,536 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 04:50:03,536 [72] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added manifestbackfillworker stdout | 2023-01-17 04:50:03,625 [72] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:10:50.626417 namespacegcworker stdout | 2023-01-17 04:50:03,937 [74] [DEBUG] [__main__] Starting namespace GC worker builder stdout | 2023-01-17 04:50:04,035 [60] [DEBUG] [__main__] Asking to start build manager with lifecycle "ephemeral" builder stdout | 2023-01-17 04:50:04,035 [60] [DEBUG] [__main__] Will pass buildman hostname quayregistry-quay-builder-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443 to builders for websocket connection builder stdout | 2023-01-17 04:50:04,035 [60] [DEBUG] [__main__] Starting build manager with lifecycle "ephemeral" builder stdout | 2023-01-17 04:50:04,035 [60] [DEBUG] [buildman.server] Initializing the lifecycle manager builder stdout | 2023-01-17 04:50:04,035 [60] [DEBUG] [buildman.manager.ephemeral] calling orchestrator_from_config builder stdout | 2023-01-17 04:50:04,035 [60] [DEBUG] [buildman.orchestrator] attempting to create orchestrator redis with kwargs {'host': 'quayregistry-quay-redis', 'password': '', 'skip_keyspace_event_setup': False, 'ssl': False, 'orchestrator_prefix': 'buildman/production/', 'canceller_only': False} builder stdout | 2023-01-17 04:50:04,036 [60] [DEBUG] [buildman.orchestrator] creating pubsub with key buildman/production/orchestrator_events gunicorn-secscan stdout | 2023-01-17 04:50:04,035 [69] [INFO] [data.registry_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:50:04,035 [69] [INFO] [data.registry_model] Using registry model `` gunicorn-secscan stdout | 2023-01-17 04:50:04,036 [69] [INFO] [data.registry_model] =============================== buildlogsarchiver stdout | 2023-01-17 04:50:04,127 [61] [INFO] [apscheduler.scheduler] Added job "ArchiveBuildLogsWorker._archive_redis_buildlogs" to job store "default" buildlogsarchiver stdout | 2023-01-17 04:50:04,127 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:50:04,128 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:27.226424+00:00 (in 23.098364 seconds) namespacegcworker stdout | 2023-01-17 04:50:04,228 [74] [DEBUG] [workers.worker] Scheduling worker. namespacegcworker stdout | 2023-01-17 04:50:04,229 [74] [INFO] [apscheduler.scheduler] Scheduler started securityworker stdout | 2023-01-17 04:50:04,228 [87] [INFO] [apscheduler.scheduler] Added job "SecurityWorker._index_in_scanner" to job store "default" securityworker stdout | 2023-01-17 04:50:04,229 [87] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:09.230038 securityworker stdout | 2023-01-17 04:50:04,229 [87] [INFO] [apscheduler.scheduler] Added job "SecurityWorker._index_recent_manifests_in_scanner" to job store "default" securityworker stdout | 2023-01-17 04:50:04,229 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:04,229 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:09.230038+00:00 (in 5.000525 seconds) namespacegcworker stdout | 2023-01-17 04:50:04,333 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:50:04,333 [74] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added namespacegcworker stdout | 2023-01-17 04:50:04,336 [74] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:59.337413 builder stdout | 2023-01-17 04:50:04,525 [60] [DEBUG] [buildman.manager.ephemeral] setting on_key_change callbacks for job expiry, cancel builder stdout | 2023-01-17 04:50:04,525 [60] [DEBUG] [buildman.orchestrator] watching key: buildman/production/building/ builder stdout | 2023-01-17 04:50:04,525 [60] [DEBUG] [buildman.orchestrator] watching key: buildman/production/cancel/ builder stdout | 2023-01-17 04:50:04,525 [60] [DEBUG] [buildman.server] Initializing the gRPC server builder stdout | 2023-01-17 04:50:04,530 [60] [DEBUG] [buildman.server] Starting the gRPC server... manifestbackfillworker stdout | 2023-01-17 04:50:04,633 [72] [INFO] [apscheduler.scheduler] Added job "ManifestBackfillWorker._backfill_manifests" to job store "default" builder stdout | 2023-01-17 04:50:04,725 [60] [DEBUG] [buildman.server] Starting the build manager... builder stdout | 2023-01-17 04:50:04,725 [60] [DEBUG] [buildman.manager.ephemeral] Initializing work checker builder stdout | 2023-01-17 04:50:04,725 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics manifestbackfillworker stdout | 2023-01-17 04:50:04,636 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 04:50:04,636 [72] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:50.626417+00:00 (in 1245.989926 seconds) repositorygcworker stdout | 2023-01-17 04:50:04,727 [85] [DEBUG] [__main__] Starting repository GC worker repositorygcworker stdout | 2023-01-17 04:50:04,925 [85] [DEBUG] [workers.worker] Scheduling worker. repositorygcworker stdout | 2023-01-17 04:50:04,926 [85] [INFO] [apscheduler.scheduler] Scheduler started builder stdout | 2023-01-17 04:50:05,025 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:50:05,025 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:50:05,025 [85] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added repositorygcworker stdout | 2023-01-17 04:50:05,029 [85] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:39.030814 gunicorn-secscan stdout | 2023-01-17 04:50:05,131 [69] [DEBUG] [app] Loading default config. gunicorn-secscan stdout | 2023-01-17 04:50:05,132 [69] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml gunicorn-secscan stdout | 2023-01-17 04:50:05,331 [69] [DEBUG] [app] Loaded config gunicorn-secscan stdout | 2023-01-17 04:50:05,332 [69] [INFO] [util.ipresolver] Loading AWS IP ranges from disk gunicorn-secscan stdout | 2023-01-17 04:50:05,435 [69] [DEBUG] [util.ipresolver] Building AWS IP ranges globalpromstats stdout | 2023-01-17 04:50:05,636 [67] [DEBUG] [workers.worker] Scheduling worker. globalpromstats stdout | 2023-01-17 04:50:05,725 [67] [INFO] [apscheduler.scheduler] Scheduler started repositorygcworker stdout | 2023-01-17 04:50:05,736 [85] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" repositorygcworker stdout | 2023-01-17 04:50:05,737 [85] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:52:04.738062 repositorygcworker stdout | 2023-01-17 04:50:05,737 [85] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" repositorygcworker stdout | 2023-01-17 04:50:05,737 [85] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:48.738458 repositorygcworker stdout | 2023-01-17 04:50:05,737 [85] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" repositorygcworker stdout | 2023-01-17 04:50:05,825 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:50:05,825 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:39.030814+00:00 (in 33.205327 seconds) globalpromstats stdout | 2023-01-17 04:50:05,825 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 04:50:05,825 [67] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added globalpromstats stdout | 2023-01-17 04:50:05,826 [67] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:03:02.826956 namespacegcworker stdout | 2023-01-17 04:50:05,930 [74] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" namespacegcworker stdout | 2023-01-17 04:50:05,931 [74] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:54:15.931994 namespacegcworker stdout | 2023-01-17 04:50:05,931 [74] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" namespacegcworker stdout | 2023-01-17 04:50:05,931 [74] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:47.932490 namespacegcworker stdout | 2023-01-17 04:50:05,931 [74] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" namespacegcworker stdout | 2023-01-17 04:50:05,936 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:50:05,936 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:47.932490+00:00 (in 41.996025 seconds) chunkcleanupworker stdout | 2023-01-17 04:50:06,039 [62] [DEBUG] [__main__] Swift storage not detected; sleeping globalpromstats stdout | 2023-01-17 04:50:06,236 [67] [INFO] [apscheduler.scheduler] Added job "GlobalPrometheusStatsWorker._try_report_stats" to job store "default" globalpromstats stdout | 2023-01-17 04:50:06,237 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 04:50:06,237 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:02.826956+00:00 (in 776.589480 seconds) logrotateworker stdout | 2023-01-17 04:50:06,429 [71] [DEBUG] [__main__] Action log rotation worker not enabled; skipping gunicorn-secscan stdout | 2023-01-17 04:50:06,435 [69] [DEBUG] [util.ipresolver] Finished building AWS IP ranges queuecleanupworker stdout | 2023-01-17 04:50:06,642 [78] [DEBUG] [workers.worker] Scheduling worker. queuecleanupworker stdout | 2023-01-17 04:50:06,643 [78] [INFO] [apscheduler.scheduler] Scheduler started exportactionlogsworker stdout | 2023-01-17 04:50:06,725 [65] [DEBUG] [__main__] Starting export action logs worker queuecleanupworker stdout | 2023-01-17 04:50:06,729 [78] [DEBUG] [apscheduler.scheduler] Looking for jobs to run queuecleanupworker stdout | 2023-01-17 04:50:06,729 [78] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added queuecleanupworker stdout | 2023-01-17 04:50:06,729 [78] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 10:19:19.730781 repositoryactioncounter stdout | 2023-01-17 04:50:06,826 [84] [DEBUG] [workers.worker] Scheduling worker. repositoryactioncounter stdout | 2023-01-17 04:50:06,827 [84] [INFO] [apscheduler.scheduler] Scheduler started repositoryactioncounter stdout | 2023-01-17 04:50:06,827 [84] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositoryactioncounter stdout | 2023-01-17 04:50:06,827 [84] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added exportactionlogsworker stdout | 2023-01-17 04:50:06,828 [65] [DEBUG] [workers.worker] Scheduling worker. repositoryactioncounter stdout | 2023-01-17 04:50:06,827 [84] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 07:48:26.828623 exportactionlogsworker stdout | 2023-01-17 04:50:06,828 [65] [INFO] [apscheduler.scheduler] Scheduler started exportactionlogsworker stdout | 2023-01-17 04:50:06,829 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:50:06,829 [65] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added exportactionlogsworker stdout | 2023-01-17 04:50:06,829 [65] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:48.830274 storagereplication stdout | 2023-01-17 04:50:06,941 [89] [DEBUG] [__main__] Full storage replication disabled; skipping gunicorn-registry stdout | 2023-01-17 04:50:07,125 [68] [INFO] [data.registry_model] =============================== gunicorn-registry stdout | 2023-01-17 04:50:07,125 [68] [INFO] [data.registry_model] Using registry model `` gunicorn-registry stdout | 2023-01-17 04:50:07,126 [68] [INFO] [data.registry_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:50:07,130 [69] [DEBUG] [data.database] Configuring database gunicorn-secscan stdout | 2023-01-17 04:50:07,132 [69] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-secscan stdout | 2023-01-17 04:50:07,132 [69] [INFO] [data.secscan_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:50:07,132 [69] [INFO] [data.secscan_model] Using split secscan model: `[]` gunicorn-secscan stdout | 2023-01-17 04:50:07,133 [69] [INFO] [data.secscan_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:50:07,133 [69] [DEBUG] [data.logs_model] Configuring log lodel gunicorn-secscan stdout | 2023-01-17 04:50:07,133 [69] [INFO] [data.logs_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:50:07,133 [69] [INFO] [data.logs_model] Using logs model `` gunicorn-secscan stdout | 2023-01-17 04:50:07,133 [69] [INFO] [data.logs_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:50:07,230 [69] [DEBUG] [__config__] Starting secscan gunicorn with 2 workers and gevent worker class repositoryactioncounter stdout | 2023-01-17 04:50:07,235 [84] [INFO] [apscheduler.scheduler] Added job "RepositoryActionCountWorker._run_counting" to job store "default" repositoryactioncounter stdout | 2023-01-17 04:50:07,235 [84] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositoryactioncounter stdout | 2023-01-17 04:50:07,235 [84] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 07:48:26.828623+00:00 (in 10699.592721 seconds) queuecleanupworker stdout | 2023-01-17 04:50:07,329 [78] [INFO] [apscheduler.scheduler] Added job "QueueCleanupWorker._cleanup_queue" to job store "default" queuecleanupworker stdout | 2023-01-17 04:50:07,329 [78] [DEBUG] [apscheduler.scheduler] Looking for jobs to run queuecleanupworker stdout | 2023-01-17 04:50:07,329 [78] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 10:19:19.730781+00:00 (in 19752.400882 seconds) gunicorn-registry stdout | 2023-01-17 04:50:07,539 [68] [DEBUG] [app] Loading default config. gunicorn-registry stdout | 2023-01-17 04:50:07,539 [68] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml gunicorn-web stdout | 2023-01-17 04:50:07,547 [70] [INFO] [data.registry_model] =============================== gunicorn-web stdout | 2023-01-17 04:50:07,547 [70] [INFO] [data.registry_model] Using registry model `` gunicorn-web stdout | 2023-01-17 04:50:07,548 [70] [INFO] [data.registry_model] =============================== nginx stdout | 2023/01/17 04:50:07 [crit] 100#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.131.0.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.131.0.22:8443" nginx stdout | 2023/01/17 04:50:07 [crit] 100#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.131.0.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.131.0.22:8443" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /health/instance HTTP/2.0" 502 157 "-" "kube-probe/1.24" (0.000 47 0.000 : 0.000) gunicorn-registry stdout | 2023-01-17 04:50:07,627 [68] [DEBUG] [app] Loaded config gunicorn-registry stdout | 2023-01-17 04:50:07,628 [68] [INFO] [util.ipresolver] Loading AWS IP ranges from disk gunicorn-registry stdout | 2023-01-17 04:50:07,635 [68] [DEBUG] [util.ipresolver] Building AWS IP ranges exportactionlogsworker stdout | 2023-01-17 04:50:07,640 [65] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" exportactionlogsworker stdout | 2023-01-17 04:50:07,640 [65] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:51:44.641497 exportactionlogsworker stdout | 2023-01-17 04:50:07,640 [65] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" exportactionlogsworker stdout | 2023-01-17 04:50:07,640 [65] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:24.641816 exportactionlogsworker stdout | 2023-01-17 04:50:07,641 [65] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" exportactionlogsworker stdout | 2023-01-17 04:50:07,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:50:07,641 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:24.641816+00:00 (in 17.000593 seconds) notificationworker stdout | 2023-01-17 04:50:07,733 [76] [DEBUG] [workers.worker] Scheduling worker. notificationworker stdout | 2023-01-17 04:50:07,733 [76] [INFO] [apscheduler.scheduler] Scheduler started notificationworker stdout | 2023-01-17 04:50:07,733 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:07,734 [76] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added notificationworker stdout | 2023-01-17 04:50:07,734 [76] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:08.735066 gcworker stdout | 2023-01-17 04:50:07,846 [66] [DEBUG] [workers.worker] Scheduling worker. gcworker stdout | 2023-01-17 04:50:07,847 [66] [INFO] [apscheduler.scheduler] Scheduler started gcworker stdout | 2023-01-17 04:50:07,847 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:50:07,847 [66] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added gcworker stdout | 2023-01-17 04:50:07,847 [66] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:35.848675 gunicorn-registry stdout | 2023-01-17 04:50:07,927 [68] [DEBUG] [util.ipresolver] Finished building AWS IP ranges notificationworker stdout | 2023-01-17 04:50:07,933 [76] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" notificationworker stdout | 2023-01-17 04:50:07,934 [76] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:53:56.935055 notificationworker stdout | 2023-01-17 04:50:07,934 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:07,934 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:08.735066+00:00 (in 0.800811 seconds) notificationworker stdout | 2023-01-17 04:50:07,934 [76] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" notificationworker stdout | 2023-01-17 04:50:07,934 [76] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:43.935629 notificationworker stdout | 2023-01-17 04:50:07,934 [76] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" notificationworker stdout | 2023-01-17 04:50:07,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:07,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:08.735066+00:00 (in 0.800019 seconds) gunicorn-web stdout | 2023-01-17 04:50:07,939 [70] [DEBUG] [app] Loading default config. gunicorn-web stdout | 2023-01-17 04:50:07,939 [70] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml gunicorn-web stdout | 2023-01-17 04:50:07,952 [70] [DEBUG] [app] Loaded config gunicorn-web stdout | 2023-01-17 04:50:07,952 [70] [INFO] [util.ipresolver] Loading AWS IP ranges from disk gunicorn-web stdout | 2023-01-17 04:50:08,028 [70] [DEBUG] [util.ipresolver] Building AWS IP ranges gcworker stdout | 2023-01-17 04:50:08,038 [66] [INFO] [apscheduler.scheduler] Added job "GarbageCollectionWorker._garbage_collection_repos" to job store "default" gcworker stdout | 2023-01-17 04:50:08,038 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:50:08,038 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:35.848675+00:00 (in 27.810382 seconds) expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,043 [64] [DEBUG] [__main__] Starting expired app specific token GC worker blobuploadcleanupworker stdout | 2023-01-17 04:50:08,044 [59] [DEBUG] [workers.worker] Scheduling worker. blobuploadcleanupworker stdout | 2023-01-17 04:50:08,045 [59] [INFO] [apscheduler.scheduler] Scheduler started blobuploadcleanupworker stdout | 2023-01-17 04:50:08,045 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 04:50:08,045 [59] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added blobuploadcleanupworker stdout | 2023-01-17 04:50:08,045 [59] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:52:45.046803 expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,055 [64] [DEBUG] [__main__] Found expiration window: 1d expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,056 [64] [DEBUG] [workers.worker] Scheduling worker. expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,057 [64] [INFO] [apscheduler.scheduler] Scheduler started expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,125 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,125 [64] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:14:09.126527 expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,125 [64] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added gunicorn-registry stdout | 2023-01-17 04:50:08,128 [68] [DEBUG] [data.database] Configuring database gunicorn-registry stdout | 2023-01-17 04:50:08,129 [68] [INFO] [data.database] Connection pooling enabled for postgresql; stale timeout: None; max connection count: None gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [INFO] [data.secscan_model] =============================== gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [INFO] [data.secscan_model] Using split secscan model: `[]` gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [INFO] [data.secscan_model] =============================== gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [DEBUG] [data.logs_model] Configuring log lodel gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [INFO] [data.logs_model] =============================== gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [INFO] [data.logs_model] Using logs model `` gunicorn-registry stdout | 2023-01-17 04:50:08,130 [68] [INFO] [data.logs_model] =============================== blobuploadcleanupworker stdout | 2023-01-17 04:50:08,160 [59] [INFO] [apscheduler.scheduler] Added job "BlobUploadCleanupWorker._try_cleanup_uploads" to job store "default" blobuploadcleanupworker stdout | 2023-01-17 04:50:08,160 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 04:50:08,160 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:45.046803+00:00 (in 156.886143 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:50:08,228 [86] [DEBUG] [__main__] Starting security scanning notification worker gunicorn-web stdout | 2023-01-17 04:50:08,230 [70] [DEBUG] [util.ipresolver] Finished building AWS IP ranges securityscanningnotificationworker stdout | 2023-01-17 04:50:08,240 [86] [DEBUG] [workers.worker] Scheduling worker. securityscanningnotificationworker stdout | 2023-01-17 04:50:08,240 [86] [INFO] [apscheduler.scheduler] Scheduler started securityscanningnotificationworker stdout | 2023-01-17 04:50:08,241 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:50:08,241 [86] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added securityscanningnotificationworker stdout | 2023-01-17 04:50:08,241 [86] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:20.242198 expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,242 [64] [INFO] [apscheduler.scheduler] Added job "ExpiredAppSpecificTokenWorker._gc_expired_tokens" to job store "default" expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,243 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 04:50:08,243 [64] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:09.126527+00:00 (in 1440.883447 seconds) gunicorn-web stdout | 2023-01-17 04:50:08,264 [70] [DEBUG] [data.database] Configuring database gunicorn-web stdout | 2023-01-17 04:50:08,265 [70] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:08,265 [70] [INFO] [data.secscan_model] =============================== gunicorn-web stdout | 2023-01-17 04:50:08,265 [70] [INFO] [data.secscan_model] Using split secscan model: `[]` gunicorn-web stdout | 2023-01-17 04:50:08,266 [70] [INFO] [data.secscan_model] =============================== gunicorn-web stdout | 2023-01-17 04:50:08,266 [70] [DEBUG] [data.logs_model] Configuring log lodel gunicorn-web stdout | 2023-01-17 04:50:08,266 [70] [INFO] [data.logs_model] =============================== gunicorn-web stdout | 2023-01-17 04:50:08,266 [70] [INFO] [data.logs_model] Using logs model `` gunicorn-web stdout | 2023-01-17 04:50:08,266 [70] [INFO] [data.logs_model] =============================== securityscanningnotificationworker stdout | 2023-01-17 04:50:08,332 [86] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" securityscanningnotificationworker stdout | 2023-01-17 04:50:08,332 [86] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:51:39.333860 securityscanningnotificationworker stdout | 2023-01-17 04:50:08,333 [86] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" securityscanningnotificationworker stdout | 2023-01-17 04:50:08,333 [86] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:25.334138 securityscanningnotificationworker stdout | 2023-01-17 04:50:08,333 [86] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" securityscanningnotificationworker stdout | 2023-01-17 04:50:08,333 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:50:08,333 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:20.242198+00:00 (in 11.908660 seconds) gunicorn-registry stdout | 2023-01-17 04:50:08,409 [68] [DEBUG] [__config__] Starting registry gunicorn with 8 workers and gevent worker class notificationworker stdout | 2023-01-17 04:50:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:08 UTC)" (scheduled at 2023-01-17 04:50:08.735066+00:00) notificationworker stdout | 2023-01-17 04:50:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 8, 735894), True, datetime.datetime(2023, 1, 17, 4, 50, 8, 735894), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:08,736 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:18.735066+00:00 (in 9.999655 seconds) notificationworker stdout | 2023-01-17 04:50:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:08,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:08,751 [70] [DEBUG] [__config__] Starting web gunicorn with 4 workers and gevent worker class securityworker stdout | 2023-01-17 04:50:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:09 UTC)" (scheduled at 2023-01-17 04:50:09.230038+00:00) securityworker stdout | 2023-01-17 04:50:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:22.837365+00:00 (in 13.607013 seconds) securityworker stdout | 2023-01-17 04:50:09,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:50:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:50:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:09,232 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:09,232 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:09,234 [87] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 securityworker stdout | 2023-01-17 04:50:09,246 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:09,248 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:09,254 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:09,254 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:50:09,255 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:39 UTC)" executed successfully builder stdout | 2023-01-17 04:50:15,035 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:15,046 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:15,047 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:15,052 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:18,737 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:18,737 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:28.735066+00:00 (in 9.997769 seconds) notificationworker stdout | 2023-01-17 04:50:18,737 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:28 UTC)" (scheduled at 2023-01-17 04:50:18.735066+00:00) notificationworker stdout | 2023-01-17 04:50:18,737 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:18,738 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 18, 737761), True, datetime.datetime(2023, 1, 17, 4, 50, 18, 737761), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:18,746 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:18,746 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:18,746 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:50:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:50:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:20 UTC)" (scheduled at 2023-01-17 04:50:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:50:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:50:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 20, 243189), True, datetime.datetime(2023, 1, 17, 4, 50, 20, 243189), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:50:20,244 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:25.334138+00:00 (in 5.091594 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:50:20,252 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:50:20,252 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:50:20,252 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:22,627 [241] [DEBUG] [app] Starting request: urn:request:e60c4b81-85d4-4388-85af-002a116fc6c3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:50:22,632 [241] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:50:22,648 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:22,650 [242] [DEBUG] [app] Starting request: urn:request:a19481f5-f43e-4e90-b918-6a9244938800 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,652 [242] [DEBUG] [app] Ending request: urn:request:a19481f5-f43e-4e90-b918-6a9244938800 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a19481f5-f43e-4e90-b918-6a9244938800', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 162 0.004) gunicorn-registry stdout | 2023-01-17 04:50:22,652 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:22,652 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:22,654 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:22,656 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:22,658 [244] [DEBUG] [app] Starting request: urn:request:e2223eda-3246-41fd-aac3-ee84fab62a2d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:22,659 [244] [DEBUG] [app] Ending request: urn:request:e2223eda-3246-41fd-aac3-ee84fab62a2d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e2223eda-3246-41fd-aac3-ee84fab62a2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 159 0.003) gunicorn-web stdout | 2023-01-17 04:50:22,660 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:22,660 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:22,662 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 50, 22, 661227), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 50, 22, 661258), 'quay', None, datetime.datetime(2023, 1, 16, 4, 50, 22, 661276), datetime.datetime(2023, 1, 10, 4, 50, 22, 661285), None]) gunicorn-web stdout | 2023-01-17 04:50:22,671 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:22,675 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:22,677 [241] [DEBUG] [app] Ending request: urn:request:e60c4b81-85d4-4388-85af-002a116fc6c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e60c4b81-85d4-4388-85af-002a116fc6c3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:22,678 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:22,678 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.054 47 0.054) gunicorn-web stdout | 2023-01-17 04:50:22,682 [239] [DEBUG] [app] Starting request: urn:request:eb868240-c27b-49e3-9e8b-b2e130175e32 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:50:22,687 [239] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:50:22,703 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:22,704 [242] [DEBUG] [app] Starting request: urn:request:b0ae0725-cb8d-43d6-a095-bbaa3092ca26 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,705 [242] [DEBUG] [app] Ending request: urn:request:b0ae0725-cb8d-43d6-a095-bbaa3092ca26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0ae0725-cb8d-43d6-a095-bbaa3092ca26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,705 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:50:22,705 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:22,707 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:22,709 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:22,709 [244] [DEBUG] [app] Starting request: urn:request:39af807a-0502-46fd-8655-5b4dbb17175b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:22,710 [244] [DEBUG] [app] Ending request: urn:request:39af807a-0502-46fd-8655-5b4dbb17175b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39af807a-0502-46fd-8655-5b4dbb17175b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:22,710 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:50:22,710 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:22,712 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 50, 22, 711507), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 50, 22, 711532), 'quay', None, datetime.datetime(2023, 1, 16, 4, 50, 22, 711546), datetime.datetime(2023, 1, 10, 4, 50, 22, 711555), None]) gunicorn-web stdout | 2023-01-17 04:50:22,720 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:22,725 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:22,727 [239] [DEBUG] [app] Ending request: urn:request:eb868240-c27b-49e3-9e8b-b2e130175e32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb868240-c27b-49e3-9e8b-b2e130175e32', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:22,728 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:22,728 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.048 47 0.048) securityworker stdout | 2023-01-17 04:50:22,838 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:39.230038+00:00 (in 16.391556 seconds) securityworker stdout | 2023-01-17 04:50:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:52 UTC)" (scheduled at 2023-01-17 04:50:22.837365+00:00) securityworker stdout | 2023-01-17 04:50:22,839 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:22,839 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:22,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:22,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:22,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:22,852 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:50:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:50:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:24 UTC)" (scheduled at 2023-01-17 04:50:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:50:24,643 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:50:24,643 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:24 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:50:24,643 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:48.830274+00:00 (in 24.188044 seconds) builder stdout | 2023-01-17 04:50:25,062 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:25,071 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:25,071 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:25,077 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:50:25,336 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:50:25,336 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:20.242198+00:00 (in 54.905858 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:50:25,336 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:25 UTC)" (scheduled at 2023-01-17 04:50:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:50:25,336 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:50:25,336 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:50:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:50:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:27 UTC)" (scheduled at 2023-01-17 04:50:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:50:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:57.226424+00:00 (in 29.999678 seconds) buildlogsarchiver stdout | 2023-01-17 04:50:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 50, 27, 227299), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:50:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:50:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:50:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:28,254 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:50:28,462 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:50:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:38.735066+00:00 (in 9.999450 seconds) notificationworker stdout | 2023-01-17 04:50:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:38 UTC)" (scheduled at 2023-01-17 04:50:28.735066+00:00) notificationworker stdout | 2023-01-17 04:50:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 28, 736113), True, datetime.datetime(2023, 1, 17, 4, 50, 28, 736113), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:38 UTC)" executed successfully servicekey stdout | 2023-01-17 04:50:28,770 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 04:50:29,559 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:50:29,642 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:50:30,842 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:50:31,559 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:50:31,665 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:50:32,160 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:50:32,852 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:50:33,171 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:50:34,060 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:50:34,560 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:50:34,760 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:50:35,087 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:35,096 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:35,096 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:35,102 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 04:50:35,342 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:50:35,344 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:50:35,345 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:50:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:50:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:35 UTC)" (scheduled at 2023-01-17 04:50:35.848675+00:00) gcworker stdout | 2023-01-17 04:50:35,850 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 04:50:35,850 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:05.848675+00:00 (in 29.999661 seconds) gcworker stdout | 2023-01-17 04:50:35,857 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721435856, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:50:35,860 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:50:35,860 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:50:35,861 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:50:35,960 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:50:36,572 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:50:37,367 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 04:50:37,463 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 04:50:37,464 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 04:50:37,466 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:50:37,626 [241] [DEBUG] [app] Starting request: urn:request:f92cc1c2-092b-4fcd-97d8-b74f88146c1e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:50:37,626 [239] [DEBUG] [app] Starting request: urn:request:748a3270-798a-4f7a-81c7-ee2a026c22de (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:50:37,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:37,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:37,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:37,638 [242] [DEBUG] [app] Starting request: urn:request:92ce1217-95d3-49cd-90cb-c5701ec93e29 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,639 [242] [DEBUG] [app] Ending request: urn:request:92ce1217-95d3-49cd-90cb-c5701ec93e29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:92ce1217-95d3-49cd-90cb-c5701ec93e29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,639 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.008 162 0.009) gunicorn-web stdout | 2023-01-17 04:50:37,640 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:50:37,640 [234] [DEBUG] [app] Starting request: urn:request:822d95c8-719c-4039-8190-ad16d33c7989 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,641 [234] [DEBUG] [app] Ending request: urn:request:822d95c8-719c-4039-8190-ad16d33c7989 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:822d95c8-719c-4039-8190-ad16d33c7989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:37,641 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.009 162 0.010) gunicorn-registry stdout | 2023-01-17 04:50:37,642 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:37,642 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:50:37,643 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-web stdout | 2023-01-17 04:50:37,644 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:37,644 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:37,645 [244] [DEBUG] [app] Starting request: urn:request:e4668625-2740-4551-b0b2-6f250245cb14 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,645 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 04:50:37,645 [244] [DEBUG] [app] Ending request: urn:request:e4668625-2740-4551-b0b2-6f250245cb14 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4668625-2740-4551-b0b2-6f250245cb14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:37,646 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:37,646 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:37,646 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:37,646 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-registry stdout | 2023-01-17 04:50:37,647 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:50:37,649 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:50:37,649 [243] [DEBUG] [app] Starting request: urn:request:5ea43884-396d-42c7-abda-c5ff808e2cc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:37,650 [243] [DEBUG] [app] Ending request: urn:request:5ea43884-396d-42c7-abda-c5ff808e2cc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ea43884-396d-42c7-abda-c5ff808e2cc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 159 0.004) gunicorn-registry stdout | 2023-01-17 04:50:37,651 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-web stdout | 2023-01-17 04:50:37,651 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:37,651 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:37,651 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:37,651 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-registry stdout | 2023-01-17 04:50:37,652 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:50:37,654 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:50:37,655 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-registry stdout | 2023-01-17 04:50:37,656 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 04:50:37,657 [239] [DEBUG] [app] Ending request: urn:request:748a3270-798a-4f7a-81c7-ee2a026c22de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:748a3270-798a-4f7a-81c7-ee2a026c22de', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:37,657 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:37,658 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.033 47 0.032) gunicorn-registry stdout | 2023-01-17 04:50:37,658 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:50:37,661 [241] [DEBUG] [app] Ending request: urn:request:f92cc1c2-092b-4fcd-97d8-b74f88146c1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f92cc1c2-092b-4fcd-97d8-b74f88146c1e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:37,661 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:37,661 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.037) gunicorn-web stdout | 2023-01-17 04:50:37,959 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:50:37,962 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:50:37,964 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:50:37,966 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:50:37,968 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:50:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:43.935629+00:00 (in 5.200004 seconds) notificationworker stdout | 2023-01-17 04:50:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:48 UTC)" (scheduled at 2023-01-17 04:50:38.735066+00:00) notificationworker stdout | 2023-01-17 04:50:38,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 38, 736051), True, datetime.datetime(2023, 1, 17, 4, 50, 38, 736051), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:38,752 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:38,752 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:38,752 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:50:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:50:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:39 UTC)" (scheduled at 2023-01-17 04:50:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:50:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:50:39,033 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 39, 31797), True, datetime.datetime(2023, 1, 17, 4, 50, 39, 31797), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:50:39,033 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:48.738458+00:00 (in 9.707263 seconds) repositorygcworker stdout | 2023-01-17 04:50:39,041 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:50:39,041 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:50:39,041 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:52.837365+00:00 (in 13.606817 seconds) securityworker stdout | 2023-01-17 04:50:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:09 UTC)" (scheduled at 2023-01-17 04:50:39.230038+00:00) securityworker stdout | 2023-01-17 04:50:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:50:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:50:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:50:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:50:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:48.735066+00:00 (in 4.798929 seconds) notificationworker stdout | 2023-01-17 04:50:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:43 UTC)" (scheduled at 2023-01-17 04:50:43.935629+00:00) notificationworker stdout | 2023-01-17 04:50:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:50:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:43 UTC)" executed successfully builder stdout | 2023-01-17 04:50:45,112 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:45,122 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:45,122 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:45,127 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:50:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:50:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:47 UTC)" (scheduled at 2023-01-17 04:50:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:50:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:50:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:47 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:50:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:59.337413+00:00 (in 11.404544 seconds) notificationworker stdout | 2023-01-17 04:50:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:58.735066+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 04:50:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:58 UTC)" (scheduled at 2023-01-17 04:50:48.735066+00:00) notificationworker stdout | 2023-01-17 04:50:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 48, 735996), True, datetime.datetime(2023, 1, 17, 4, 50, 48, 735996), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:50:48,740 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:50:48,741 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:39.030814+00:00 (in 50.289799 seconds) repositorygcworker stdout | 2023-01-17 04:50:48,741 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:48 UTC)" (scheduled at 2023-01-17 04:50:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:50:48,741 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:50:48,741 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:50:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:50:48,831 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:50:48,832 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:24.641816+00:00 (in 35.809909 seconds) exportactionlogsworker stdout | 2023-01-17 04:50:48,832 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:48 UTC)" (scheduled at 2023-01-17 04:50:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:50:48,832 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:50:48,833 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 48, 832342), True, datetime.datetime(2023, 1, 17, 4, 50, 48, 832342), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:50:48,840 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:50:48,840 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:50:48,840 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:52,626 [241] [DEBUG] [app] Starting request: urn:request:e401a11c-2706-4a3d-b5c7-093fb0db3762 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:50:52,626 [244] [DEBUG] [app] Starting request: urn:request:f799ab98-46e7-4422-96fb-d6a2652bd326 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:50:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:52,630 [244] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-registry stdout | 2023-01-17 04:50:52,631 [232] [DEBUG] [app] Starting request: urn:request:309e1b6f-c200-4e7c-8b23-f56107587941 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:52,632 [232] [DEBUG] [app] Ending request: urn:request:309e1b6f-c200-4e7c-8b23-f56107587941 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:309e1b6f-c200-4e7c-8b23-f56107587941', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:50:52,633 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:52,633 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:52,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:52,637 [243] [DEBUG] [app] Starting request: urn:request:efa03b35-9847-4fcf-9d5b-f44facc29a0a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:52,638 [243] [DEBUG] [app] Ending request: urn:request:efa03b35-9847-4fcf-9d5b-f44facc29a0a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:efa03b35-9847-4fcf-9d5b-f44facc29a0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:50:52,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:52,640 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:52,641 [232] [DEBUG] [app] Starting request: urn:request:aea41c3e-d47c-4ee3-aa12-7ec73260a151 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:52,641 [232] [DEBUG] [app] Ending request: urn:request:aea41c3e-d47c-4ee3-aa12-7ec73260a151 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aea41c3e-d47c-4ee3-aa12-7ec73260a151', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:52,642 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:50:52,642 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:52,643 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:52,645 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:52,646 [243] [DEBUG] [app] Starting request: urn:request:c95c0092-2eac-4325-92fc-fef1c16122e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:52,646 [243] [DEBUG] [app] Ending request: urn:request:c95c0092-2eac-4325-92fc-fef1c16122e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c95c0092-2eac-4325-92fc-fef1c16122e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:52,646 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:52,646 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:52,648 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 50, 52, 647797), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 50, 52, 647826), 'quay', None, datetime.datetime(2023, 1, 16, 4, 50, 52, 647845), datetime.datetime(2023, 1, 10, 4, 50, 52, 647856), None]) gunicorn-web stdout | 2023-01-17 04:50:52,649 [241] [DEBUG] [app] Ending request: urn:request:e401a11c-2706-4a3d-b5c7-093fb0db3762 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e401a11c-2706-4a3d-b5c7-093fb0db3762', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:52,649 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 04:50:52,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:50:52,656 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:52,660 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:52,662 [244] [DEBUG] [app] Ending request: urn:request:f799ab98-46e7-4422-96fb-d6a2652bd326 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f799ab98-46e7-4422-96fb-d6a2652bd326', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:52,662 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.038) gunicorn-web stdout | 2023-01-17 04:50:52,662 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:50:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:09.230038+00:00 (in 16.392157 seconds) securityworker stdout | 2023-01-17 04:50:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:22 UTC)" (scheduled at 2023-01-17 04:50:52.837365+00:00) securityworker stdout | 2023-01-17 04:50:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:22 UTC)" executed successfully builder stdout | 2023-01-17 04:50:55,138 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:55,147 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:55,147 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:55,153 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:50:57,227 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:50:57,228 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:27.226424+00:00 (in 29.998324 seconds) buildlogsarchiver stdout | 2023-01-17 04:50:57,228 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:27 UTC)" (scheduled at 2023-01-17 04:50:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:50:57,229 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 50, 57, 228513), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:50:57,236 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:50:57,236 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:50:57,236 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:58,261 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:50:58,475 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:50:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:08.735066+00:00 (in 9.999461 seconds) notificationworker stdout | 2023-01-17 04:50:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:08 UTC)" (scheduled at 2023-01-17 04:50:58.735066+00:00) notificationworker stdout | 2023-01-17 04:50:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 58, 736076), True, datetime.datetime(2023, 1, 17, 4, 50, 58, 736076), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:08 UTC)" executed successfully servicekey stdout | 2023-01-17 04:50:58,781 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:50:59,338 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:50:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:47.932490+00:00 (in 48.593767 seconds) namespacegcworker stdout | 2023-01-17 04:50:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:59 UTC)" (scheduled at 2023-01-17 04:50:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:50:59,339 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:50:59,340 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 59, 339123), True, datetime.datetime(2023, 1, 17, 4, 50, 59, 339123), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:50:59,347 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:50:59,347 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:50:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:59 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:50:59,566 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:50:59,647 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:51:00,852 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:51:01,595 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:51:01,674 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:51:02,185 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:51:02,888 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:51:03,195 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:51:04,095 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:51:04,595 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:51:04,783 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:51:05,163 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:05,172 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:05,172 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:05,178 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 04:51:05,350 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:51:05,352 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:51:05,354 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:51:05,850 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:51:05,850 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:35.848675+00:00 (in 29.998303 seconds) gcworker stdout | 2023-01-17 04:51:05,850 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:35 UTC)" (scheduled at 2023-01-17 04:51:05.848675+00:00) gcworker stdout | 2023-01-17 04:51:05,851 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721465850, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:51:05,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:51:05,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:51:05,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:51:05,983 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:51:06,607 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:51:07,403 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:51:07,470 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 04:51:07,485 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 04:51:07,497 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:51:07,625 [241] [DEBUG] [app] Starting request: urn:request:11c30a6a-0619-4ed6-bbea-fea7d7b409f5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:07,626 [243] [DEBUG] [app] Starting request: urn:request:bc7a07a0-ead8-43a8-8fa9-120d4ed9a0f6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,630 [243] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:51:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:07,631 [232] [DEBUG] [app] Starting request: urn:request:0b470790-231d-46f9-93f0-828cb11cb081 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,631 [232] [DEBUG] [app] Ending request: urn:request:0b470790-231d-46f9-93f0-828cb11cb081 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b470790-231d-46f9-93f0-828cb11cb081', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:51:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:07,636 [239] [DEBUG] [app] Starting request: urn:request:66e5fd41-c29a-468a-85ff-d628b5e91d0c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:07,637 [239] [DEBUG] [app] Ending request: urn:request:66e5fd41-c29a-468a-85ff-d628b5e91d0c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66e5fd41-c29a-468a-85ff-d628b5e91d0c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:51:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:07,642 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-registry stdout | 2023-01-17 04:51:07,643 [232] [DEBUG] [app] Starting request: urn:request:a87a9c5c-f93b-472e-bdcb-9b9a8e86a90a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,643 [232] [DEBUG] [app] Ending request: urn:request:a87a9c5c-f93b-472e-bdcb-9b9a8e86a90a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a87a9c5c-f93b-472e-bdcb-9b9a8e86a90a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,643 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:51:07,644 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:07,645 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,647 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:07,648 [244] [DEBUG] [app] Starting request: urn:request:7968dbc5-c2c3-495a-92d8-8a5b423299f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:07,648 [241] [DEBUG] [app] Ending request: urn:request:11c30a6a-0619-4ed6-bbea-fea7d7b409f5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11c30a6a-0619-4ed6-bbea-fea7d7b409f5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:07,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:07,649 [244] [DEBUG] [app] Ending request: urn:request:7968dbc5-c2c3-495a-92d8-8a5b423299f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7968dbc5-c2c3-495a-92d8-8a5b423299f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:51:07,649 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:07,649 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:51:07,650 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-web stdout | 2023-01-17 04:51:07,652 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 51, 7, 650706), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 51, 7, 650747), 'quay', None, datetime.datetime(2023, 1, 16, 4, 51, 7, 650771), datetime.datetime(2023, 1, 10, 4, 51, 7, 650786), None]) gunicorn-registry stdout | 2023-01-17 04:51:07,653 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:51:07,656 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:51:07,658 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 04:51:07,660 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-registry stdout | 2023-01-17 04:51:07,661 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:51:07,664 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:51:07,666 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:51:07,669 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 04:51:07,670 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-registry stdout | 2023-01-17 04:51:07,671 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:51:07,672 [243] [DEBUG] [app] Ending request: urn:request:bc7a07a0-ead8-43a8-8fa9-120d4ed9a0f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc7a07a0-ead8-43a8-8fa9-120d4ed9a0f6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:07,672 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.047 47 0.048) gunicorn-web stdout | 2023-01-17 04:51:07,672 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:07,966 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:51:07,968 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:51:07,971 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:51:07,973 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:51:07,976 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:51:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:18.735066+00:00 (in 9.999453 seconds) notificationworker stdout | 2023-01-17 04:51:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:18 UTC)" (scheduled at 2023-01-17 04:51:08.735066+00:00) notificationworker stdout | 2023-01-17 04:51:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:08,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 8, 736093), True, datetime.datetime(2023, 1, 17, 4, 51, 8, 736093), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:22.837365+00:00 (in 13.606768 seconds) securityworker stdout | 2023-01-17 04:51:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:39 UTC)" (scheduled at 2023-01-17 04:51:09.230038+00:00) securityworker stdout | 2023-01-17 04:51:09,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:51:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:51:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:51:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:39 UTC)" executed successfully builder stdout | 2023-01-17 04:51:15,188 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:15,197 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:15,197 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:15,203 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:28.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 04:51:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:28 UTC)" (scheduled at 2023-01-17 04:51:18.735066+00:00) notificationworker stdout | 2023-01-17 04:51:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 18, 736011), True, datetime.datetime(2023, 1, 17, 4, 51, 18, 736011), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:51:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:51:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:25.334138+00:00 (in 5.091483 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:51:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:20 UTC)" (scheduled at 2023-01-17 04:51:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:51:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:51:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 20, 243095), True, datetime.datetime(2023, 1, 17, 4, 51, 20, 243095), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:51:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:51:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:51:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:51:22,625 [241] [DEBUG] [app] Starting request: urn:request:924fc57f-8e0e-474b-9e18-cbe0f8f098d7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:22,625 [239] [DEBUG] [app] Starting request: urn:request:f1654d50-9701-476c-b7d5-643d8d173429 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:22,631 [232] [DEBUG] [app] Starting request: urn:request:5a481c7c-34bd-4982-9673-878d6b860891 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:22,631 [237] [DEBUG] [app] Starting request: urn:request:8a6d8d26-3caf-45c4-9b19-da2cefdac2ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:22,631 [232] [DEBUG] [app] Ending request: urn:request:5a481c7c-34bd-4982-9673-878d6b860891 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a481c7c-34bd-4982-9673-878d6b860891', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:51:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:51:22,632 [237] [DEBUG] [app] Ending request: urn:request:8a6d8d26-3caf-45c4-9b19-da2cefdac2ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a6d8d26-3caf-45c4-9b19-da2cefdac2ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 162 0.003) gunicorn-registry stdout | 2023-01-17 04:51:22,633 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:22,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:22,636 [239] [DEBUG] [app] Starting request: urn:request:ea463442-8309-4f1d-bc1e-540becad6a39 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:22,636 [239] [DEBUG] [app] Ending request: urn:request:ea463442-8309-4f1d-bc1e-540becad6a39 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea463442-8309-4f1d-bc1e-540becad6a39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:22,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:22,638 [244] [DEBUG] [app] Starting request: urn:request:b18c4ba7-cef7-46b6-bbf3-c94e498a1384 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:22,638 [244] [DEBUG] [app] Ending request: urn:request:b18c4ba7-cef7-46b6-bbf3-c94e498a1384 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b18c4ba7-cef7-46b6-bbf3-c94e498a1384', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:22,639 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:22,639 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:22,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:22,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:22,646 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:22,649 [239] [DEBUG] [app] Ending request: urn:request:f1654d50-9701-476c-b7d5-643d8d173429 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1654d50-9701-476c-b7d5-643d8d173429', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:22,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:22,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.025) gunicorn-web stdout | 2023-01-17 04:51:22,652 [241] [DEBUG] [app] Ending request: urn:request:924fc57f-8e0e-474b-9e18-cbe0f8f098d7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:924fc57f-8e0e-474b-9e18-cbe0f8f098d7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:22,652 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 04:51:22,652 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:51:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:39.230038+00:00 (in 16.392173 seconds) securityworker stdout | 2023-01-17 04:51:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:52 UTC)" (scheduled at 2023-01-17 04:51:22.837365+00:00) securityworker stdout | 2023-01-17 04:51:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:51:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:51:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:44.641497+00:00 (in 19.999174 seconds) exportactionlogsworker stdout | 2023-01-17 04:51:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:24 UTC)" (scheduled at 2023-01-17 04:51:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:51:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:51:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:24 UTC)" executed successfully builder stdout | 2023-01-17 04:51:25,204 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:25,213 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:25,213 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:25,219 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:51:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:51:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:39.333860+00:00 (in 13.999266 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:51:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:25 UTC)" (scheduled at 2023-01-17 04:51:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:51:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:51:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:51:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:51:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:57.226424+00:00 (in 29.999509 seconds) buildlogsarchiver stdout | 2023-01-17 04:51:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:57 UTC)" (scheduled at 2023-01-17 04:51:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:51:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 51, 27, 227374), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:51:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:51:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:51:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:28,297 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:51:28,511 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:51:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:38.735066+00:00 (in 9.999360 seconds) notificationworker stdout | 2023-01-17 04:51:28,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:38 UTC)" (scheduled at 2023-01-17 04:51:28.735066+00:00) notificationworker stdout | 2023-01-17 04:51:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 28, 736176), True, datetime.datetime(2023, 1, 17, 4, 51, 28, 736176), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:38 UTC)" executed successfully servicekey stdout | 2023-01-17 04:51:28,817 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 04:51:29,574 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:51:29,668 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:51:30,861 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:51:31,631 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:51:31,692 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:51:32,221 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:51:32,918 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:51:33,230 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:51:34,106 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:51:34,613 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:51:34,805 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:51:35,229 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:35,238 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:35,238 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:35,244 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 04:51:35,356 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:51:35,359 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:51:35,361 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:51:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:51:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:05.848675+00:00 (in 29.999491 seconds) gcworker stdout | 2023-01-17 04:51:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:05 UTC)" (scheduled at 2023-01-17 04:51:35.848675+00:00) gcworker stdout | 2023-01-17 04:51:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721495849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:51:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:51:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:51:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:51:36,015 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:51:36,643 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:51:37,439 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:51:37,481 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 04:51:37,520 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 04:51:37,532 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:51:37,626 [241] [DEBUG] [app] Starting request: urn:request:9f6e1802-de6a-4c00-929c-fc0e10982391 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:37,626 [239] [DEBUG] [app] Starting request: urn:request:d9875e14-b643-43c6-aba8-ebdd0e558b2f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:37,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:37,632 [237] [DEBUG] [app] Starting request: urn:request:5c7280cc-f324-4be6-8deb-86bb1dcff1cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:37,632 [240] [DEBUG] [app] Starting request: urn:request:a90e34b6-9045-41bc-aa2d-55a64d290ae5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:37,632 [237] [DEBUG] [app] Ending request: urn:request:5c7280cc-f324-4be6-8deb-86bb1dcff1cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c7280cc-f324-4be6-8deb-86bb1dcff1cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:37,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:51:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:51:37,633 [240] [DEBUG] [app] Ending request: urn:request:a90e34b6-9045-41bc-aa2d-55a64d290ae5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a90e34b6-9045-41bc-aa2d-55a64d290ae5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:51:37,634 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:37,634 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:37,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,635 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:37,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:37,637 [244] [DEBUG] [app] Starting request: urn:request:0cf840a9-2f71-40bd-a650-d7f62c32d74f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:37,638 [244] [DEBUG] [app] Ending request: urn:request:0cf840a9-2f71-40bd-a650-d7f62c32d74f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0cf840a9-2f71-40bd-a650-d7f62c32d74f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:37,638 [241] [DEBUG] [app] Starting request: urn:request:62c77aa3-f3f6-4350-9593-ece42398861e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.003) gunicorn-web stdout | 2023-01-17 04:51:37,638 [241] [DEBUG] [app] Ending request: urn:request:62c77aa3-f3f6-4350-9593-ece42398861e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:62c77aa3-f3f6-4350-9593-ece42398861e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:37,639 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:37,639 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:37,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:37,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:37,644 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:37,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:37,649 [241] [DEBUG] [app] Ending request: urn:request:9f6e1802-de6a-4c00-929c-fc0e10982391 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f6e1802-de6a-4c00-929c-fc0e10982391', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:37,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:37,649 [239] [DEBUG] [app] Ending request: urn:request:d9875e14-b643-43c6-aba8-ebdd0e558b2f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9875e14-b643-43c6-aba8-ebdd0e558b2f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:37,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:37,650 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:51:37,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-registry stdout | 2023-01-17 04:51:37,657 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:51:37,659 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:51:37,662 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:51:37,665 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:51:37,667 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:51:37,670 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:51:37,672 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:51:37,676 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:51:37,678 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:51:37,972 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:51:37,974 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:51:37,977 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:51:37,979 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:51:37,982 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:51:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:43.935629+00:00 (in 5.200048 seconds) notificationworker stdout | 2023-01-17 04:51:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:48 UTC)" (scheduled at 2023-01-17 04:51:38.735066+00:00) notificationworker stdout | 2023-01-17 04:51:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 38, 735991), True, datetime.datetime(2023, 1, 17, 4, 51, 38, 735991), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:51:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:51:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:48.738458+00:00 (in 9.707209 seconds) repositorygcworker stdout | 2023-01-17 04:51:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:39 UTC)" (scheduled at 2023-01-17 04:51:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:51:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:51:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 39, 31669), True, datetime.datetime(2023, 1, 17, 4, 51, 39, 31669), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:51:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:51:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:51:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:52.837365+00:00 (in 13.606716 seconds) securityworker stdout | 2023-01-17 04:51:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:09 UTC)" (scheduled at 2023-01-17 04:51:39.230038+00:00) securityworker stdout | 2023-01-17 04:51:39,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:51:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:51:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:51:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:51:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:51:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:20.242198+00:00 (in 40.907839 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:51:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:39 UTC)" (scheduled at 2023-01-17 04:51:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:51:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 51, 39, 334714), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:51:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 39, 334714), True, datetime.datetime(2023, 1, 17, 4, 51, 39, 334714), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:51:39,344 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 39, 334714), True, datetime.datetime(2023, 1, 17, 4, 51, 39, 334714), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 4, 51, 39, 334714), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:51:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:51:39,346 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:51:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:48.735066+00:00 (in 4.798948 seconds) notificationworker stdout | 2023-01-17 04:51:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:43 UTC)" (scheduled at 2023-01-17 04:51:43.935629+00:00) notificationworker stdout | 2023-01-17 04:51:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:51:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:51:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:51:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:48.830274+00:00 (in 4.188271 seconds) exportactionlogsworker stdout | 2023-01-17 04:51:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:44 UTC)" (scheduled at 2023-01-17 04:51:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 04:51:44,643 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 51, 44, 642474), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:51:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 44, 642474), True, datetime.datetime(2023, 1, 17, 4, 51, 44, 642474), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:51:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 44, 642474), True, datetime.datetime(2023, 1, 17, 4, 51, 44, 642474), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 4, 51, 44, 642474), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:51:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:51:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:44 UTC)" executed successfully builder stdout | 2023-01-17 04:51:45,246 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:45,256 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:45,256 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:45,261 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:51:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:51:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:59.337413+00:00 (in 11.404452 seconds) namespacegcworker stdout | 2023-01-17 04:51:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:47 UTC)" (scheduled at 2023-01-17 04:51:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:51:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:51:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:51:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:58.735066+00:00 (in 9.999452 seconds) notificationworker stdout | 2023-01-17 04:51:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:58 UTC)" (scheduled at 2023-01-17 04:51:48.735066+00:00) notificationworker stdout | 2023-01-17 04:51:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 48, 736034), True, datetime.datetime(2023, 1, 17, 4, 51, 48, 736034), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:51:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:51:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:04.738062+00:00 (in 15.999144 seconds) repositorygcworker stdout | 2023-01-17 04:51:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:48 UTC)" (scheduled at 2023-01-17 04:51:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:51:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:51:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:51:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:51:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:51:48,831 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:24.641816+00:00 (in 35.811002 seconds) exportactionlogsworker stdout | 2023-01-17 04:51:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:48 UTC)" (scheduled at 2023-01-17 04:51:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:51:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:51:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 48, 831290), True, datetime.datetime(2023, 1, 17, 4, 51, 48, 831290), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:51:48,839 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:51:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:51:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:51:52,625 [243] [DEBUG] [app] Starting request: urn:request:018e2c44-e266-4ed6-ae69-311b1b5c7825 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:52,625 [241] [DEBUG] [app] Starting request: urn:request:00f85015-69a1-4bed-b4e8-e951b79bd8d6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:51:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:52,631 [240] [DEBUG] [app] Starting request: urn:request:04ae2104-8026-42ed-8f87-89d783fcd9bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:52,631 [242] [DEBUG] [app] Starting request: urn:request:2588662f-8b69-49d2-9ff7-62f9701962bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:52,631 [240] [DEBUG] [app] Ending request: urn:request:04ae2104-8026-42ed-8f87-89d783fcd9bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04ae2104-8026-42ed-8f87-89d783fcd9bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:51:52,632 [242] [DEBUG] [app] Ending request: urn:request:2588662f-8b69-49d2-9ff7-62f9701962bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2588662f-8b69-49d2-9ff7-62f9701962bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:51:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:52,636 [239] [DEBUG] [app] Starting request: urn:request:fa93f02a-71b4-4d6b-aca0-a2aa8c5db341 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:52,636 [244] [DEBUG] [app] Starting request: urn:request:418da37c-7f31-4579-bec9-5ef31fd854d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:52,636 [239] [DEBUG] [app] Ending request: urn:request:fa93f02a-71b4-4d6b-aca0-a2aa8c5db341 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa93f02a-71b4-4d6b-aca0-a2aa8c5db341', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:52,637 [244] [DEBUG] [app] Ending request: urn:request:418da37c-7f31-4579-bec9-5ef31fd854d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:418da37c-7f31-4579-bec9-5ef31fd854d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:51:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:52,648 [243] [DEBUG] [app] Ending request: urn:request:018e2c44-e266-4ed6-ae69-311b1b5c7825 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:018e2c44-e266-4ed6-ae69-311b1b5c7825', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:52,648 [241] [DEBUG] [app] Ending request: urn:request:00f85015-69a1-4bed-b4e8-e951b79bd8d6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:00f85015-69a1-4bed-b4e8-e951b79bd8d6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 04:51:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:09.230038+00:00 (in 16.392153 seconds) securityworker stdout | 2023-01-17 04:51:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:22 UTC)" (scheduled at 2023-01-17 04:51:52.837365+00:00) securityworker stdout | 2023-01-17 04:51:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:52,856 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:52,858 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:52,858 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:22 UTC)" executed successfully builder stdout | 2023-01-17 04:51:55,272 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:55,280 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:55,280 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:55,286 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:51:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:51:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:27.226424+00:00 (in 29.999456 seconds) buildlogsarchiver stdout | 2023-01-17 04:51:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:27 UTC)" (scheduled at 2023-01-17 04:51:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:51:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 51, 57, 227376), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:51:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:51:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:51:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:58,303 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:51:58,546 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:51:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:08.735066+00:00 (in 9.999480 seconds) notificationworker stdout | 2023-01-17 04:51:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:08 UTC)" (scheduled at 2023-01-17 04:51:58.735066+00:00) notificationworker stdout | 2023-01-17 04:51:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 58, 736034), True, datetime.datetime(2023, 1, 17, 4, 51, 58, 736034), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:08 UTC)" executed successfully servicekey stdout | 2023-01-17 04:51:58,853 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:51:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:51:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:47.932490+00:00 (in 48.594593 seconds) namespacegcworker stdout | 2023-01-17 04:51:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:59 UTC)" (scheduled at 2023-01-17 04:51:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:51:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:51:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 59, 338295), True, datetime.datetime(2023, 1, 17, 4, 51, 59, 338295), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:51:59,349 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:51:59,349 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:51:59,349 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:59 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:51:59,609 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:51:59,675 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:52:00,896 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:52:01,667 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:52:01,709 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:52:02,231 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:52:02,952 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:52:03,255 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:52:04,117 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:52:04,649 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} repositorygcworker stdout | 2023-01-17 04:52:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:52:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:39.030814+00:00 (in 34.292294 seconds) repositorygcworker stdout | 2023-01-17 04:52:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:57:04 UTC)" (scheduled at 2023-01-17 04:52:04.738062+00:00) repositorygcworker stdout | 2023-01-17 04:52:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 52, 4, 738895), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:52:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 52, 4, 738895), True, datetime.datetime(2023, 1, 17, 4, 52, 4, 738895), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:52:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 52, 4, 738895), True, datetime.datetime(2023, 1, 17, 4, 52, 4, 738895), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 4, 52, 4, 738895), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:52:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:52:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:57:04 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 04:52:04,829 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:52:05,296 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:05,305 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:05,305 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:05,311 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 04:52:05,365 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:52:05,367 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:52:05,369 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:52:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:52:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:35.848675+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 04:52:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:35 UTC)" (scheduled at 2023-01-17 04:52:05.848675+00:00) gcworker stdout | 2023-01-17 04:52:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721525849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:52:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:52:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:52:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:52:06,021 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:52:06,654 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:52:07,475 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:52:07,516 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:52:07,550 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:52:07,554 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:52:07,625 [241] [DEBUG] [app] Starting request: urn:request:c30ceae8-1aac-4e8f-bae3-08e034c45cdb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:07,625 [239] [DEBUG] [app] Starting request: urn:request:69fb07be-2bbd-4982-bfb9-b66b851cc618 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:07,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:07,630 [232] [DEBUG] [app] Starting request: urn:request:4e418175-5023-4524-92e3-02e680bfacf8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:07,631 [232] [DEBUG] [app] Ending request: urn:request:4e418175-5023-4524-92e3-02e680bfacf8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e418175-5023-4524-92e3-02e680bfacf8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:52:07,631 [236] [DEBUG] [app] Starting request: urn:request:5e02715d-6fe8-440c-9132-724fbccfaf70 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:07,632 [236] [DEBUG] [app] Ending request: urn:request:5e02715d-6fe8-440c-9132-724fbccfaf70 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e02715d-6fe8-440c-9132-724fbccfaf70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:07,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:52:07,633 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:07,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:07,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:07,635 [243] [DEBUG] [app] Starting request: urn:request:0c1ebace-2d46-4610-9008-ef27d59382f0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:07,636 [243] [DEBUG] [app] Ending request: urn:request:0c1ebace-2d46-4610-9008-ef27d59382f0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c1ebace-2d46-4610-9008-ef27d59382f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:07,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:07,637 [243] [DEBUG] [app] Starting request: urn:request:6d3edffa-381c-4dd7-90b1-d814f2128799 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:07,637 [243] [DEBUG] [app] Ending request: urn:request:6d3edffa-381c-4dd7-90b1-d814f2128799 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d3edffa-381c-4dd7-90b1-d814f2128799', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:07,647 [241] [DEBUG] [app] Ending request: urn:request:c30ceae8-1aac-4e8f-bae3-08e034c45cdb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c30ceae8-1aac-4e8f-bae3-08e034c45cdb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:07,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 04:52:07,647 [239] [DEBUG] [app] Ending request: urn:request:69fb07be-2bbd-4982-bfb9-b66b851cc618 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69fb07be-2bbd-4982-bfb9-b66b851cc618', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-registry stdout | 2023-01-17 04:52:07,663 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:52:07,667 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:52:07,670 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:52:07,672 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:52:07,675 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:52:07,677 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:52:07,679 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:52:07,682 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:52:07,684 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:52:07,979 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:52:07,982 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:52:07,984 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:52:07,986 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:52:07,989 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:52:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:18.735066+00:00 (in 9.999485 seconds) notificationworker stdout | 2023-01-17 04:52:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:18 UTC)" (scheduled at 2023-01-17 04:52:08.735066+00:00) notificationworker stdout | 2023-01-17 04:52:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 8, 736013), True, datetime.datetime(2023, 1, 17, 4, 52, 8, 736013), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:22.837365+00:00 (in 13.606875 seconds) securityworker stdout | 2023-01-17 04:52:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:39 UTC)" (scheduled at 2023-01-17 04:52:09.230038+00:00) securityworker stdout | 2023-01-17 04:52:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:52:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:52:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:52:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:52:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:52:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:52:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:39 UTC)" executed successfully builder stdout | 2023-01-17 04:52:15,321 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:15,330 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:15,330 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:15,336 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:28.735066+00:00 (in 9.999468 seconds) notificationworker stdout | 2023-01-17 04:52:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:28 UTC)" (scheduled at 2023-01-17 04:52:18.735066+00:00) notificationworker stdout | 2023-01-17 04:52:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 18, 736024), True, datetime.datetime(2023, 1, 17, 4, 52, 18, 736024), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:52:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:52:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:25.334138+00:00 (in 5.091461 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:52:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:20 UTC)" (scheduled at 2023-01-17 04:52:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:52:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:52:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 20, 243070), True, datetime.datetime(2023, 1, 17, 4, 52, 20, 243070), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:52:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:52:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:52:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:22,625 [244] [DEBUG] [app] Starting request: urn:request:545d97c3-2b81-401f-98b2-f2a68c1bea7a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:22,625 [243] [DEBUG] [app] Starting request: urn:request:acdead4d-5644-40de-aa23-9178e953fbfc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:22,630 [242] [DEBUG] [app] Starting request: urn:request:cf1f01a5-3e96-400e-8e56-afaa37420a1d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,630 [242] [DEBUG] [app] Ending request: urn:request:cf1f01a5-3e96-400e-8e56-afaa37420a1d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf1f01a5-3e96-400e-8e56-afaa37420a1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,630 [232] [DEBUG] [app] Starting request: urn:request:095c4a71-0ff2-4185-892b-6e8b4ad9abef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:52:22,631 [232] [DEBUG] [app] Ending request: urn:request:095c4a71-0ff2-4185-892b-6e8b4ad9abef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:095c4a71-0ff2-4185-892b-6e8b4ad9abef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:22,635 [239] [DEBUG] [app] Starting request: urn:request:67dc74f1-9c80-4454-a884-8e28de3b05d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:22,635 [239] [DEBUG] [app] Ending request: urn:request:67dc74f1-9c80-4454-a884-8e28de3b05d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:67dc74f1-9c80-4454-a884-8e28de3b05d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:22,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:22,636 [241] [DEBUG] [app] Starting request: urn:request:47bb9d2a-8614-4661-be91-449267a4fe9e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:22,637 [241] [DEBUG] [app] Ending request: urn:request:47bb9d2a-8614-4661-be91-449267a4fe9e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47bb9d2a-8614-4661-be91-449267a4fe9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:22,647 [243] [DEBUG] [app] Ending request: urn:request:acdead4d-5644-40de-aa23-9178e953fbfc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:acdead4d-5644-40de-aa23-9178e953fbfc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:52:22,648 [244] [DEBUG] [app] Ending request: urn:request:545d97c3-2b81-401f-98b2-f2a68c1bea7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:545d97c3-2b81-401f-98b2-f2a68c1bea7a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 04:52:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) securityworker stdout | 2023-01-17 04:52:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:39.230038+00:00 (in 16.392202 seconds) securityworker stdout | 2023-01-17 04:52:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:52 UTC)" (scheduled at 2023-01-17 04:52:22.837365+00:00) securityworker stdout | 2023-01-17 04:52:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:52:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:52:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:48.830274+00:00 (in 24.187924 seconds) exportactionlogsworker stdout | 2023-01-17 04:52:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:24 UTC)" (scheduled at 2023-01-17 04:52:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:52:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:52:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:52:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:52:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:20.242198+00:00 (in 54.907598 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:52:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:25 UTC)" (scheduled at 2023-01-17 04:52:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:52:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:52:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:25 UTC)" executed successfully builder stdout | 2023-01-17 04:52:25,346 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:25,355 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:25,355 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:25,361 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:52:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:52:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:57.226424+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 04:52:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:57 UTC)" (scheduled at 2023-01-17 04:52:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:52:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 52, 27, 227298), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:52:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:52:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:52:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:28,335 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:52:28,582 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:52:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:38.735066+00:00 (in 9.999494 seconds) notificationworker stdout | 2023-01-17 04:52:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:38 UTC)" (scheduled at 2023-01-17 04:52:28.735066+00:00) notificationworker stdout | 2023-01-17 04:52:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 28, 736028), True, datetime.datetime(2023, 1, 17, 4, 52, 28, 736028), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:38 UTC)" executed successfully servicekey stdout | 2023-01-17 04:52:28,889 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 04:52:29,645 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:52:29,710 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:52:30,932 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:52:31,703 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:52:31,745 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:52:32,238 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:52:32,974 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:52:33,290 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:52:34,153 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:52:34,684 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:52:34,864 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:52:35,363 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:35,372 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:35,372 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-secscan stdout | 2023-01-17 04:52:35,373 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:52:35,375 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:52:35,378 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:52:35,379 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:52:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:52:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:05.848675+00:00 (in 29.999505 seconds) gcworker stdout | 2023-01-17 04:52:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:05 UTC)" (scheduled at 2023-01-17 04:52:35.848675+00:00) gcworker stdout | 2023-01-17 04:52:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721555849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:52:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:52:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:52:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:52:36,044 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:52:36,675 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:52:37,480 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:52:37,551 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:52:37,555 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:52:37,573 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:52:37,625 [239] [DEBUG] [app] Starting request: urn:request:091e1323-46df-4458-8277-88f89aaa3a86 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:37,625 [244] [DEBUG] [app] Starting request: urn:request:1c48ec36-e401-46f9-8b70-fa00193f7bb5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:37,630 [237] [DEBUG] [app] Starting request: urn:request:fb911de6-161c-4755-a685-8b462a857164 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:37,630 [240] [DEBUG] [app] Starting request: urn:request:8c85f918-fe12-4c55-b08a-417cde123400 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:37,630 [237] [DEBUG] [app] Ending request: urn:request:fb911de6-161c-4755-a685-8b462a857164 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fb911de6-161c-4755-a685-8b462a857164', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:52:37,631 [240] [DEBUG] [app] Ending request: urn:request:8c85f918-fe12-4c55-b08a-417cde123400 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c85f918-fe12-4c55-b08a-417cde123400', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:52:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:37,636 [244] [DEBUG] [app] Starting request: urn:request:0d405d76-c16d-439e-bfa1-6c8705a3123d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:37,636 [244] [DEBUG] [app] Ending request: urn:request:0d405d76-c16d-439e-bfa1-6c8705a3123d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d405d76-c16d-439e-bfa1-6c8705a3123d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:37,636 [243] [DEBUG] [app] Starting request: urn:request:0029de5f-57bd-4c5e-b921-c004fb61e001 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:37,637 [243] [DEBUG] [app] Ending request: urn:request:0029de5f-57bd-4c5e-b921-c004fb61e001 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0029de5f-57bd-4c5e-b921-c004fb61e001', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:37,647 [244] [DEBUG] [app] Ending request: urn:request:1c48ec36-e401-46f9-8b70-fa00193f7bb5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c48ec36-e401-46f9-8b70-fa00193f7bb5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:52:37,648 [239] [DEBUG] [app] Ending request: urn:request:091e1323-46df-4458-8277-88f89aaa3a86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:091e1323-46df-4458-8277-88f89aaa3a86', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:37,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:52:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 04:52:37,670 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:52:37,673 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:52:37,675 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:52:37,678 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:52:37,680 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:52:37,683 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:52:37,686 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:52:37,688 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:52:37,690 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:52:37,984 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:52:37,988 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:52:37,990 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:52:37,993 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:52:37,995 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:52:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:43.935629+00:00 (in 5.200028 seconds) notificationworker stdout | 2023-01-17 04:52:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:48 UTC)" (scheduled at 2023-01-17 04:52:38.735066+00:00) notificationworker stdout | 2023-01-17 04:52:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 38, 735986), True, datetime.datetime(2023, 1, 17, 4, 52, 38, 735986), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:52:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:52:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:48.738458+00:00 (in 9.707189 seconds) repositorygcworker stdout | 2023-01-17 04:52:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:39 UTC)" (scheduled at 2023-01-17 04:52:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:52:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:52:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 39, 31688), True, datetime.datetime(2023, 1, 17, 4, 52, 39, 31688), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:52:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:52:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:52:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:52.837365+00:00 (in 13.606875 seconds) securityworker stdout | 2023-01-17 04:52:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:09 UTC)" (scheduled at 2023-01-17 04:52:39.230038+00:00) securityworker stdout | 2023-01-17 04:52:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:52:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:52:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:52:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:52:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:52:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:52:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:52:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:48.735066+00:00 (in 4.798962 seconds) notificationworker stdout | 2023-01-17 04:52:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:43 UTC)" (scheduled at 2023-01-17 04:52:43.935629+00:00) notificationworker stdout | 2023-01-17 04:52:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:52:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:43 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 04:52:45,047 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 04:52:45,047 [59] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2023-01-17 04:52:45 UTC)" (scheduled at 2023-01-17 04:52:45.046803+00:00) blobuploadcleanupworker stdout | 2023-01-17 04:52:45,047 [59] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 04:52:45,048 [59] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ... blobuploadcleanupworker stdout | 2023-01-17 04:52:45,048 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:45.046803+00:00 (in 3599.999640 seconds) blobuploadcleanupworker stdout | 2023-01-17 04:52:45,048 [59] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2023-01-17 04:52:45,048 [59] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 04:52:45,048 [59] [DEBUG] [__main__] Performing blob upload cleanup blobuploadcleanupworker stdout | 2023-01-17 04:52:45,049 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 15, 4, 52, 45, 48728), 1, 0]) blobuploadcleanupworker stdout | 2023-01-17 04:52:45,056 [59] [DEBUG] [__main__] No additional stale blob uploads found blobuploadcleanupworker stdout | 2023-01-17 04:52:45,056 [59] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 04:52:45,057 [59] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 04:52:45,057 [59] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2023-01-17 04:52:45,057 [59] [DEBUG] [util.locking] Released lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 04:52:45,057 [59] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2023-01-17 05:52:45 UTC)" executed successfully builder stdout | 2023-01-17 04:52:45,389 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:45,397 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:45,397 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:45,403 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:52:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:52:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:59.337413+00:00 (in 11.404417 seconds) namespacegcworker stdout | 2023-01-17 04:52:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:47 UTC)" (scheduled at 2023-01-17 04:52:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:52:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:52:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:52:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:58.735066+00:00 (in 9.999468 seconds) notificationworker stdout | 2023-01-17 04:52:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:58 UTC)" (scheduled at 2023-01-17 04:52:48.735066+00:00) notificationworker stdout | 2023-01-17 04:52:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 48, 736019), True, datetime.datetime(2023, 1, 17, 4, 52, 48, 736019), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:52:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:52:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:39.030814+00:00 (in 50.291879 seconds) repositorygcworker stdout | 2023-01-17 04:52:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:48 UTC)" (scheduled at 2023-01-17 04:52:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:52:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:52:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:52:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:52:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:52:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:24.641816+00:00 (in 35.811075 seconds) exportactionlogsworker stdout | 2023-01-17 04:52:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:48 UTC)" (scheduled at 2023-01-17 04:52:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:52:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:52:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 48, 831158), True, datetime.datetime(2023, 1, 17, 4, 52, 48, 831158), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:52:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:52:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:52:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:52,625 [239] [DEBUG] [app] Starting request: urn:request:3b4cc60d-59df-47a1-a86c-c5eac41af135 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:52,625 [243] [DEBUG] [app] Starting request: urn:request:e065f2a9-acec-4d36-a5a2-2bafdb2a9ff1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:52:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:52,630 [242] [DEBUG] [app] Starting request: urn:request:6bae46c3-e857-474c-aadd-d94839ea194f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:52,630 [240] [DEBUG] [app] Starting request: urn:request:d90e0d3d-c232-4826-8215-22415edcc8b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:52,630 [242] [DEBUG] [app] Ending request: urn:request:6bae46c3-e857-474c-aadd-d94839ea194f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6bae46c3-e857-474c-aadd-d94839ea194f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 04:52:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:52:52,631 [240] [DEBUG] [app] Ending request: urn:request:d90e0d3d-c232-4826-8215-22415edcc8b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d90e0d3d-c232-4826-8215-22415edcc8b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:52:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:52:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:52,635 [239] [DEBUG] [app] Starting request: urn:request:034f308c-e99a-4156-acd8-f5ed055eafdf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:52,635 [239] [DEBUG] [app] Ending request: urn:request:034f308c-e99a-4156-acd8-f5ed055eafdf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:034f308c-e99a-4156-acd8-f5ed055eafdf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:52,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:52,637 [241] [DEBUG] [app] Starting request: urn:request:cab6b626-9c32-403c-979e-4c68b3a00ca1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:52,638 [241] [DEBUG] [app] Ending request: urn:request:cab6b626-9c32-403c-979e-4c68b3a00ca1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cab6b626-9c32-403c-979e-4c68b3a00ca1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:52,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:52,647 [243] [DEBUG] [app] Ending request: urn:request:e065f2a9-acec-4d36-a5a2-2bafdb2a9ff1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e065f2a9-acec-4d36-a5a2-2bafdb2a9ff1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:52:52,649 [239] [DEBUG] [app] Ending request: urn:request:3b4cc60d-59df-47a1-a86c-c5eac41af135 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3b4cc60d-59df-47a1-a86c-c5eac41af135', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:52,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) securityworker stdout | 2023-01-17 04:52:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:09.230038+00:00 (in 16.392218 seconds) securityworker stdout | 2023-01-17 04:52:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:22 UTC)" (scheduled at 2023-01-17 04:52:52.837365+00:00) securityworker stdout | 2023-01-17 04:52:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:22 UTC)" executed successfully builder stdout | 2023-01-17 04:52:55,403 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:55,412 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:55,412 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:55,418 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:52:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:52:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:27.226424+00:00 (in 29.999499 seconds) buildlogsarchiver stdout | 2023-01-17 04:52:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:27 UTC)" (scheduled at 2023-01-17 04:52:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:52:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 52, 57, 227293), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:52:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:52:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:52:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:58,371 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:52:58,620 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:52:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:08.735066+00:00 (in 9.999474 seconds) notificationworker stdout | 2023-01-17 04:52:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:08 UTC)" (scheduled at 2023-01-17 04:52:58.735066+00:00) notificationworker stdout | 2023-01-17 04:52:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 58, 736018), True, datetime.datetime(2023, 1, 17, 4, 52, 58, 736018), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:08 UTC)" executed successfully servicekey stdout | 2023-01-17 04:52:58,924 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:52:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:52:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:47.932490+00:00 (in 48.594621 seconds) namespacegcworker stdout | 2023-01-17 04:52:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:59 UTC)" (scheduled at 2023-01-17 04:52:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:52:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:52:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 59, 338278), True, datetime.datetime(2023, 1, 17, 4, 52, 59, 338278), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:52:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:52:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:52:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:59 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:52:59,681 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:52:59,718 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:53:00,960 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:53:01,711 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:53:01,780 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:53:02,260 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:53:02,979 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:53:03,296 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:53:04,175 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:53:04,701 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:53:04,900 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:53:05,380 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:53:05,382 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:53:05,385 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:53:05,425 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:05,434 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:05,434 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:05,440 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:53:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:53:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:35.848675+00:00 (in 29.999447 seconds) gcworker stdout | 2023-01-17 04:53:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:35 UTC)" (scheduled at 2023-01-17 04:53:05.848675+00:00) gcworker stdout | 2023-01-17 04:53:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721585849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:53:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:53:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:53:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:53:06,059 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:53:06,712 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:53:07,494 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:53:07,586 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:53:07,588 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:53:07,608 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:53:07,625 [244] [DEBUG] [app] Starting request: urn:request:21ea1772-6a93-4990-b7a6-c556cf27c981 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:07,625 [239] [DEBUG] [app] Starting request: urn:request:ee97eba4-1ea6-4914-9ac5-118f7aa94e76 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:07,629 [240] [DEBUG] [app] Starting request: urn:request:ea56d918-64b8-4660-9fd6-979f8b62e466 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,630 [240] [DEBUG] [app] Ending request: urn:request:ea56d918-64b8-4660-9fd6-979f8b62e466 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea56d918-64b8-4660-9fd6-979f8b62e466', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,630 [232] [DEBUG] [app] Starting request: urn:request:9f5b00fe-140c-4d52-9ed6-e9c6eb6b08f7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:53:07,630 [232] [DEBUG] [app] Ending request: urn:request:9f5b00fe-140c-4d52-9ed6-e9c6eb6b08f7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9f5b00fe-140c-4d52-9ed6-e9c6eb6b08f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:07,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:53:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:07,635 [243] [DEBUG] [app] Starting request: urn:request:04249606-f9fe-4b27-b98b-1446eaa2424d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:07,635 [243] [DEBUG] [app] Ending request: urn:request:04249606-f9fe-4b27-b98b-1446eaa2424d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:04249606-f9fe-4b27-b98b-1446eaa2424d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:07,636 [241] [DEBUG] [app] Starting request: urn:request:ef432313-4fd9-45bf-a761-eedece5ea824 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:07,637 [241] [DEBUG] [app] Ending request: urn:request:ef432313-4fd9-45bf-a761-eedece5ea824 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef432313-4fd9-45bf-a761-eedece5ea824', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:07,646 [244] [DEBUG] [app] Ending request: urn:request:21ea1772-6a93-4990-b7a6-c556cf27c981 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21ea1772-6a93-4990-b7a6-c556cf27c981', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:53:07,647 [239] [DEBUG] [app] Ending request: urn:request:ee97eba4-1ea6-4914-9ac5-118f7aa94e76 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee97eba4-1ea6-4914-9ac5-118f7aa94e76', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-registry stdout | 2023-01-17 04:53:07,676 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:53:07,679 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:53:07,682 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:53:07,684 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:53:07,687 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:53:07,689 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:53:07,692 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:53:07,694 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:53:07,697 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:53:07,991 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:53:07,993 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:53:07,996 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:53:07,999 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:53:08,001 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:53:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:18.735066+00:00 (in 9.999482 seconds) notificationworker stdout | 2023-01-17 04:53:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:18 UTC)" (scheduled at 2023-01-17 04:53:08.735066+00:00) notificationworker stdout | 2023-01-17 04:53:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 8, 735999), True, datetime.datetime(2023, 1, 17, 4, 53, 8, 735999), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:22.837365+00:00 (in 13.606880 seconds) securityworker stdout | 2023-01-17 04:53:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:39 UTC)" (scheduled at 2023-01-17 04:53:09.230038+00:00) securityworker stdout | 2023-01-17 04:53:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:53:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:53:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:53:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:39 UTC)" executed successfully builder stdout | 2023-01-17 04:53:15,450 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:15,459 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:15,459 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:15,465 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:28.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 04:53:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:28 UTC)" (scheduled at 2023-01-17 04:53:18.735066+00:00) notificationworker stdout | 2023-01-17 04:53:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 18, 735970), True, datetime.datetime(2023, 1, 17, 4, 53, 18, 735970), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:53:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:53:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:25.334138+00:00 (in 5.091425 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:53:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:20 UTC)" (scheduled at 2023-01-17 04:53:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:53:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:53:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 20, 243120), True, datetime.datetime(2023, 1, 17, 4, 53, 20, 243120), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:53:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:53:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:53:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:53:22,625 [243] [DEBUG] [app] Starting request: urn:request:66e311aa-35e1-4f29-aa94-a918d015d5c5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,626 [239] [DEBUG] [app] Starting request: urn:request:121c1963-e1af-4ffc-b2db-57bf6c0b44d8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:22,630 [232] [DEBUG] [app] Starting request: urn:request:fea765ea-9e64-4242-a9e6-70676dd5d47d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:22,631 [232] [DEBUG] [app] Ending request: urn:request:fea765ea-9e64-4242-a9e6-70676dd5d47d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fea765ea-9e64-4242-a9e6-70676dd5d47d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:53:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:53:22,631 [242] [DEBUG] [app] Starting request: urn:request:f790d3cc-8156-49bb-833c-06a9dd68d103 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:53:22,631 [242] [DEBUG] [app] Ending request: urn:request:f790d3cc-8156-49bb-833c-06a9dd68d103 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f790d3cc-8156-49bb-833c-06a9dd68d103', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:53:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:22,635 [244] [DEBUG] [app] Starting request: urn:request:e5b8d470-cd4c-43cf-b24b-81ecc3b96aa0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,636 [244] [DEBUG] [app] Ending request: urn:request:e5b8d470-cd4c-43cf-b24b-81ecc3b96aa0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5b8d470-cd4c-43cf-b24b-81ecc3b96aa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:53:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:22,637 [244] [DEBUG] [app] Starting request: urn:request:b8ee734f-3217-4631-896b-041fd58425fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,637 [244] [DEBUG] [app] Ending request: urn:request:b8ee734f-3217-4631-896b-041fd58425fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8ee734f-3217-4631-896b-041fd58425fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:22,647 [243] [DEBUG] [app] Ending request: urn:request:66e311aa-35e1-4f29-aa94-a918d015d5c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66e311aa-35e1-4f29-aa94-a918d015d5c5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:53:22,648 [239] [DEBUG] [app] Ending request: urn:request:121c1963-e1af-4ffc-b2db-57bf6c0b44d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:121c1963-e1af-4ffc-b2db-57bf6c0b44d8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:22,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 04:53:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:53:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:39.230038+00:00 (in 16.392249 seconds) securityworker stdout | 2023-01-17 04:53:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:52 UTC)" (scheduled at 2023-01-17 04:53:22.837365+00:00) securityworker stdout | 2023-01-17 04:53:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:53:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:53:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:48.830274+00:00 (in 24.187935 seconds) exportactionlogsworker stdout | 2023-01-17 04:53:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:24 UTC)" (scheduled at 2023-01-17 04:53:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:53:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:53:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:53:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:53:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:20.242198+00:00 (in 54.907618 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:53:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:25 UTC)" (scheduled at 2023-01-17 04:53:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:53:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:53:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:25 UTC)" executed successfully builder stdout | 2023-01-17 04:53:25,475 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:25,484 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:25,484 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:25,490 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:53:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:53:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:57.226424+00:00 (in 29.999513 seconds) buildlogsarchiver stdout | 2023-01-17 04:53:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:57 UTC)" (scheduled at 2023-01-17 04:53:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:53:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 53, 27, 227327), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:53:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:53:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:53:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:28,408 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:53:28,656 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:53:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:38.735066+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 04:53:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:38 UTC)" (scheduled at 2023-01-17 04:53:28.735066+00:00) notificationworker stdout | 2023-01-17 04:53:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 28, 736044), True, datetime.datetime(2023, 1, 17, 4, 53, 28, 736044), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:38 UTC)" executed successfully servicekey stdout | 2023-01-17 04:53:28,933 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 04:53:29,717 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:53:29,753 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:53:30,969 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:53:31,747 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:53:31,816 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:53:32,282 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:53:33,015 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:53:33,304 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:53:34,210 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:53:34,716 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:53:34,930 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:53:35,388 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:53:35,390 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:53:35,392 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:53:35,500 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:35,509 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:35,509 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:35,515 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:53:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:53:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:05.848675+00:00 (in 29.999500 seconds) gcworker stdout | 2023-01-17 04:53:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:05 UTC)" (scheduled at 2023-01-17 04:53:35.848675+00:00) gcworker stdout | 2023-01-17 04:53:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721615849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:53:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:53:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:53:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:53:36,095 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:53:36,732 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:53:37,530 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:53:37,617 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:53:37,624 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:53:37,625 [239] [DEBUG] [app] Starting request: urn:request:6bb271e5-e378-4f9f-9d3d-0e394530d58f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:37,625 [244] [DEBUG] [app] Starting request: urn:request:cd9f3133-17ee-43ca-b0ed-dfafaec1248a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:37,630 [240] [DEBUG] [app] Starting request: urn:request:fa559b51-9708-481c-b53e-8b791a430b5f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,630 [232] [DEBUG] [app] Starting request: urn:request:2540041a-c75d-4385-bad1-f02da7a991cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,631 [240] [DEBUG] [app] Ending request: urn:request:fa559b51-9708-481c-b53e-8b791a430b5f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fa559b51-9708-481c-b53e-8b791a430b5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,631 [232] [DEBUG] [app] Ending request: urn:request:2540041a-c75d-4385-bad1-f02da7a991cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2540041a-c75d-4385-bad1-f02da7a991cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:53:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:53:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:37,636 [243] [DEBUG] [app] Starting request: urn:request:a06da000-0e0a-41bd-9dcb-e3759268bead (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:37,636 [241] [DEBUG] [app] Starting request: urn:request:658a2761-7abd-4886-909a-eaa2ba9d62ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:37,636 [243] [DEBUG] [app] Ending request: urn:request:a06da000-0e0a-41bd-9dcb-e3759268bead (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a06da000-0e0a-41bd-9dcb-e3759268bead', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:37,636 [241] [DEBUG] [app] Ending request: urn:request:658a2761-7abd-4886-909a-eaa2ba9d62ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:658a2761-7abd-4886-909a-eaa2ba9d62ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:53:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:53:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) securityscanningnotificationworker stdout | 2023-01-17 04:53:37,643 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:53:37,647 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:37,648 [244] [DEBUG] [app] Ending request: urn:request:cd9f3133-17ee-43ca-b0ed-dfafaec1248a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd9f3133-17ee-43ca-b0ed-dfafaec1248a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:53:37,653 [239] [DEBUG] [app] Ending request: urn:request:6bb271e5-e378-4f9f-9d3d-0e394530d58f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6bb271e5-e378-4f9f-9d3d-0e394530d58f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:37,653 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:53:37,653 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 04:53:37,682 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:53:37,685 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:53:37,688 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:53:37,693 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:53:37,695 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:53:37,698 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:53:37,700 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:53:37,702 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:53:37,704 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:53:37,998 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:53:38,001 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:53:38,004 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:53:38,006 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:53:38,008 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:53:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:43.935629+00:00 (in 5.200065 seconds) notificationworker stdout | 2023-01-17 04:53:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:48 UTC)" (scheduled at 2023-01-17 04:53:38.735066+00:00) notificationworker stdout | 2023-01-17 04:53:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 38, 735986), True, datetime.datetime(2023, 1, 17, 4, 53, 38, 735986), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:53:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:53:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:48.738458+00:00 (in 9.707172 seconds) repositorygcworker stdout | 2023-01-17 04:53:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:39 UTC)" (scheduled at 2023-01-17 04:53:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:53:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:53:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 39, 31764), True, datetime.datetime(2023, 1, 17, 4, 53, 39, 31764), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:53:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:53:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:53:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:52.837365+00:00 (in 13.606849 seconds) securityworker stdout | 2023-01-17 04:53:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:09 UTC)" (scheduled at 2023-01-17 04:53:39.230038+00:00) securityworker stdout | 2023-01-17 04:53:39,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:53:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:53:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:53:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:53:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:53:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:53:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:53:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:48.735066+00:00 (in 4.798964 seconds) notificationworker stdout | 2023-01-17 04:53:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:43 UTC)" (scheduled at 2023-01-17 04:53:43.935629+00:00) notificationworker stdout | 2023-01-17 04:53:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:53:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:43 UTC)" executed successfully builder stdout | 2023-01-17 04:53:45,523 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:45,532 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:45,532 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:45,537 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:53:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:53:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:59.337413+00:00 (in 11.404455 seconds) namespacegcworker stdout | 2023-01-17 04:53:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:47 UTC)" (scheduled at 2023-01-17 04:53:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:53:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:53:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:53:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:56.935055+00:00 (in 8.199508 seconds) notificationworker stdout | 2023-01-17 04:53:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:58 UTC)" (scheduled at 2023-01-17 04:53:48.735066+00:00) notificationworker stdout | 2023-01-17 04:53:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 48, 735980), True, datetime.datetime(2023, 1, 17, 4, 53, 48, 735980), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:53:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:53:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:39.030814+00:00 (in 50.291828 seconds) repositorygcworker stdout | 2023-01-17 04:53:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:48 UTC)" (scheduled at 2023-01-17 04:53:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:53:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:53:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:53:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:53:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:53:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:24.641816+00:00 (in 35.811081 seconds) exportactionlogsworker stdout | 2023-01-17 04:53:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:48 UTC)" (scheduled at 2023-01-17 04:53:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:53:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:53:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 48, 831129), True, datetime.datetime(2023, 1, 17, 4, 53, 48, 831129), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:53:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:53:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:53:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:53:52,626 [243] [DEBUG] [app] Starting request: urn:request:4fa830a7-a817-474e-8896-d32f336acc84 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:52,626 [239] [DEBUG] [app] Starting request: urn:request:14893c9e-dea5-476f-9d74-b4ae7c4325a2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:53:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:52,631 [232] [DEBUG] [app] Starting request: urn:request:f601a843-495b-4df4-8585-78130a624cca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,631 [240] [DEBUG] [app] Starting request: urn:request:6f0de4c8-d029-4dfc-9b21-17bc0e43a1b9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,631 [232] [DEBUG] [app] Ending request: urn:request:f601a843-495b-4df4-8585-78130a624cca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f601a843-495b-4df4-8585-78130a624cca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,631 [240] [DEBUG] [app] Ending request: urn:request:6f0de4c8-d029-4dfc-9b21-17bc0e43a1b9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f0de4c8-d029-4dfc-9b21-17bc0e43a1b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:53:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:53:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:52,636 [244] [DEBUG] [app] Starting request: urn:request:a0420317-fe95-41fa-8426-80d88bbab039 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:52,636 [239] [DEBUG] [app] Starting request: urn:request:67c729d0-f787-492f-939b-9e0c76f76ccc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:52,636 [239] [DEBUG] [app] Ending request: urn:request:67c729d0-f787-492f-939b-9e0c76f76ccc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:67c729d0-f787-492f-939b-9e0c76f76ccc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:52,636 [244] [DEBUG] [app] Ending request: urn:request:a0420317-fe95-41fa-8426-80d88bbab039 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0420317-fe95-41fa-8426-80d88bbab039', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:52,647 [239] [DEBUG] [app] Ending request: urn:request:14893c9e-dea5-476f-9d74-b4ae7c4325a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14893c9e-dea5-476f-9d74-b4ae7c4325a2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:52,647 [243] [DEBUG] [app] Ending request: urn:request:4fa830a7-a817-474e-8896-d32f336acc84 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4fa830a7-a817-474e-8896-d32f336acc84', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 04:53:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 04:53:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:09.230038+00:00 (in 16.392134 seconds) securityworker stdout | 2023-01-17 04:53:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:22 UTC)" (scheduled at 2023-01-17 04:53:52.837365+00:00) securityworker stdout | 2023-01-17 04:53:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:22 UTC)" executed successfully builder stdout | 2023-01-17 04:53:55,540 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:55,549 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:55,549 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:55,554 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:58.735066+00:00 (in 1.799525 seconds) notificationworker stdout | 2023-01-17 04:53:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:58:56 UTC)" (scheduled at 2023-01-17 04:53:56.935055+00:00) notificationworker stdout | 2023-01-17 04:53:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 53, 56, 935909), 'notification/%']) notificationworker stdout | 2023-01-17 04:53:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 53, 56, 935909), True, datetime.datetime(2023, 1, 17, 4, 53, 56, 935909), 0, 'notification/%']) notificationworker stdout | 2023-01-17 04:53:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 53, 56, 935909), True, datetime.datetime(2023, 1, 17, 4, 53, 56, 935909), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 4, 53, 56, 935909), 'notification/%']) notificationworker stdout | 2023-01-17 04:53:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:58:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:53:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:53:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:27.226424+00:00 (in 29.999525 seconds) buildlogsarchiver stdout | 2023-01-17 04:53:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:27 UTC)" (scheduled at 2023-01-17 04:53:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:53:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 53, 57, 227296), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:53:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:53:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:53:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:58,424 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:53:58,695 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:53:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:08.735066+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 04:53:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:08 UTC)" (scheduled at 2023-01-17 04:53:58.735066+00:00) notificationworker stdout | 2023-01-17 04:53:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 58, 735961), True, datetime.datetime(2023, 1, 17, 4, 53, 58, 735961), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:08 UTC)" executed successfully servicekey stdout | 2023-01-17 04:53:58,956 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:53:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:53:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:15.931994+00:00 (in 16.594119 seconds) namespacegcworker stdout | 2023-01-17 04:53:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:59 UTC)" (scheduled at 2023-01-17 04:53:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:53:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:53:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 59, 338298), True, datetime.datetime(2023, 1, 17, 4, 53, 59, 338298), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:53:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:53:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:53:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:59 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:53:59,753 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:53:59,760 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:54:00,988 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:54:01,783 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:54:01,837 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:54:02,317 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:54:03,051 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:54:03,317 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:54:04,216 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:54:04,752 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:54:04,965 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:54:05,396 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:54:05,398 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:54:05,400 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:54:05,561 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:05,570 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:05,570 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:05,576 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:54:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:54:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:35.848675+00:00 (in 29.999489 seconds) gcworker stdout | 2023-01-17 04:54:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:35 UTC)" (scheduled at 2023-01-17 04:54:05.848675+00:00) gcworker stdout | 2023-01-17 04:54:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721645849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:54:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:54:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:54:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:54:06,110 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:54:06,767 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:54:07,566 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 04:54:07,626 [244] [DEBUG] [app] Starting request: urn:request:91693421-ff30-4cba-96b6-476e349fa6e4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:07,626 [243] [DEBUG] [app] Starting request: urn:request:ce067f2e-189e-4e98-af84-dbf421110112 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost expiredappspecifictokenworker stdout | 2023-01-17 04:54:07,629 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 04:54:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:07,631 [242] [DEBUG] [app] Starting request: urn:request:63673325-8093-44fc-a5bb-fd98748d6ccc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,631 [232] [DEBUG] [app] Starting request: urn:request:15af1dd2-bd4d-49df-9c97-4669a6e1a4d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,631 [242] [DEBUG] [app] Ending request: urn:request:63673325-8093-44fc-a5bb-fd98748d6ccc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:63673325-8093-44fc-a5bb-fd98748d6ccc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,632 [232] [DEBUG] [app] Ending request: urn:request:15af1dd2-bd4d-49df-9c97-4669a6e1a4d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15af1dd2-bd4d-49df-9c97-4669a6e1a4d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:54:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:54:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:07,636 [239] [DEBUG] [app] Starting request: urn:request:5c1521e3-e885-41e2-a3d0-e901ae6dead4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:07,637 [241] [DEBUG] [app] Starting request: urn:request:361dd26b-e1e4-4bd4-badc-872444cc304c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:07,637 [239] [DEBUG] [app] Ending request: urn:request:5c1521e3-e885-41e2-a3d0-e901ae6dead4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5c1521e3-e885-41e2-a3d0-e901ae6dead4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:07,637 [241] [DEBUG] [app] Ending request: urn:request:361dd26b-e1e4-4bd4-badc-872444cc304c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:361dd26b-e1e4-4bd4-badc-872444cc304c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:07,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:07,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:07,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:07,649 [243] [DEBUG] [app] Ending request: urn:request:ce067f2e-189e-4e98-af84-dbf421110112 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce067f2e-189e-4e98-af84-dbf421110112', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:07,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:07,649 [244] [DEBUG] [app] Ending request: urn:request:91693421-ff30-4cba-96b6-476e349fa6e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:91693421-ff30-4cba-96b6-476e349fa6e4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:07,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 04:54:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" blobuploadcleanupworker stdout | 2023-01-17 04:54:07,659 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:54:07,678 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-registry stdout | 2023-01-17 04:54:07,688 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:54:07,691 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:54:07,693 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:54:07,699 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:54:07,703 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:54:07,706 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:54:07,708 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:54:07,711 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:54:07,713 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:54:08,004 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:54:08,007 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:54:08,010 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:54:08,012 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:54:08,015 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:54:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:18.735066+00:00 (in 9.999485 seconds) notificationworker stdout | 2023-01-17 04:54:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:18 UTC)" (scheduled at 2023-01-17 04:54:08.735066+00:00) notificationworker stdout | 2023-01-17 04:54:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 8, 736040), True, datetime.datetime(2023, 1, 17, 4, 54, 8, 736040), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:22.837365+00:00 (in 13.606888 seconds) securityworker stdout | 2023-01-17 04:54:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:39 UTC)" (scheduled at 2023-01-17 04:54:09.230038+00:00) securityworker stdout | 2023-01-17 04:54:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:54:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:54:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:54:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:39 UTC)" executed successfully builder stdout | 2023-01-17 04:54:15,586 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:15,595 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:15,595 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:15,601 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:54:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:54:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:47.932490+00:00 (in 32.000048 seconds) namespacegcworker stdout | 2023-01-17 04:54:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:59:15 UTC)" (scheduled at 2023-01-17 04:54:15.931994+00:00) namespacegcworker stdout | 2023-01-17 04:54:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 54, 15, 932819), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:54:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 54, 15, 932819), True, datetime.datetime(2023, 1, 17, 4, 54, 15, 932819), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:54:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 54, 15, 932819), True, datetime.datetime(2023, 1, 17, 4, 54, 15, 932819), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 4, 54, 15, 932819), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:54:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:54:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:59:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:54:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:28.735066+00:00 (in 9.999488 seconds) notificationworker stdout | 2023-01-17 04:54:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:28 UTC)" (scheduled at 2023-01-17 04:54:18.735066+00:00) notificationworker stdout | 2023-01-17 04:54:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 18, 736010), True, datetime.datetime(2023, 1, 17, 4, 54, 18, 736010), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:54:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:54:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:25.334138+00:00 (in 5.091483 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:54:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:20 UTC)" (scheduled at 2023-01-17 04:54:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:54:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:54:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 20, 243075), True, datetime.datetime(2023, 1, 17, 4, 54, 20, 243075), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:54:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:54:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:54:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:54:22,625 [239] [DEBUG] [app] Starting request: urn:request:105e7079-d03b-4d0c-8f2b-d2439ae01add (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:22,625 [244] [DEBUG] [app] Starting request: urn:request:7b31fa27-7ff9-49db-af61-9f5a144ee98d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:22,630 [232] [DEBUG] [app] Starting request: urn:request:d83576b8-76b6-4f10-be52-b3ade1ecf500 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,630 [242] [DEBUG] [app] Starting request: urn:request:effeb050-b367-4c94-8d54-071dc7a84d26 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,631 [232] [DEBUG] [app] Ending request: urn:request:d83576b8-76b6-4f10-be52-b3ade1ecf500 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d83576b8-76b6-4f10-be52-b3ade1ecf500', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:54:22,631 [242] [DEBUG] [app] Ending request: urn:request:effeb050-b367-4c94-8d54-071dc7a84d26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:effeb050-b367-4c94-8d54-071dc7a84d26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:54:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:54:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:22,635 [243] [DEBUG] [app] Starting request: urn:request:cb58b505-3659-4444-9e0e-ef4eb185fc03 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:22,635 [241] [DEBUG] [app] Starting request: urn:request:8b6dccde-4e91-4f3f-847c-a2b2cb6679d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:22,635 [243] [DEBUG] [app] Ending request: urn:request:cb58b505-3659-4444-9e0e-ef4eb185fc03 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cb58b505-3659-4444-9e0e-ef4eb185fc03', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:22,636 [241] [DEBUG] [app] Ending request: urn:request:8b6dccde-4e91-4f3f-847c-a2b2cb6679d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b6dccde-4e91-4f3f-847c-a2b2cb6679d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:54:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:22,646 [239] [DEBUG] [app] Ending request: urn:request:105e7079-d03b-4d0c-8f2b-d2439ae01add (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:105e7079-d03b-4d0c-8f2b-d2439ae01add', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:22,647 [244] [DEBUG] [app] Ending request: urn:request:7b31fa27-7ff9-49db-af61-9f5a144ee98d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b31fa27-7ff9-49db-af61-9f5a144ee98d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:54:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 04:54:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:39.230038+00:00 (in 16.392254 seconds) securityworker stdout | 2023-01-17 04:54:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:52 UTC)" (scheduled at 2023-01-17 04:54:22.837365+00:00) securityworker stdout | 2023-01-17 04:54:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:54:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:54:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:48.830274+00:00 (in 24.187971 seconds) exportactionlogsworker stdout | 2023-01-17 04:54:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:24 UTC)" (scheduled at 2023-01-17 04:54:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:54:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:54:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:54:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:54:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:20.242198+00:00 (in 54.907626 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:54:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:25 UTC)" (scheduled at 2023-01-17 04:54:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:54:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:54:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:25 UTC)" executed successfully builder stdout | 2023-01-17 04:54:25,611 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:25,620 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:25,620 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:25,626 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:54:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:54:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:57.226424+00:00 (in 29.999563 seconds) buildlogsarchiver stdout | 2023-01-17 04:54:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:57 UTC)" (scheduled at 2023-01-17 04:54:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:54:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 54, 27, 227202), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:54:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:54:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:54:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:28,456 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:54:28,732 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:54:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:38.735066+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 04:54:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:38 UTC)" (scheduled at 2023-01-17 04:54:28.735066+00:00) notificationworker stdout | 2023-01-17 04:54:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 28, 735954), True, datetime.datetime(2023, 1, 17, 4, 54, 28, 735954), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:38 UTC)" executed successfully servicekey stdout | 2023-01-17 04:54:28,965 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 04:54:29,760 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:54:29,780 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:54:30,999 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:54:31,819 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:54:31,866 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:54:32,349 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:54:33,082 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:54:33,352 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:54:34,235 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:54:34,771 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:54:34,981 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:54:35,403 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:54:35,405 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:54:35,407 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:54:35,636 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:35,645 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:35,645 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:35,651 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:54:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:54:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:05.848675+00:00 (in 29.999533 seconds) gcworker stdout | 2023-01-17 04:54:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:05 UTC)" (scheduled at 2023-01-17 04:54:35.848675+00:00) gcworker stdout | 2023-01-17 04:54:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721675849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:54:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:54:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:54:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:54:36,136 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:54:36,802 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:54:37,590 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 04:54:37,625 [239] [DEBUG] [app] Starting request: urn:request:44602d30-ceff-4c66-994e-32fa941659ab (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:37,625 [244] [DEBUG] [app] Starting request: urn:request:14426245-cf15-4233-829e-a9692b0e2241 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:37,630 [242] [DEBUG] [app] Starting request: urn:request:debcace4-dbe0-4f79-8f7c-216811486b19 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:37,630 [242] [DEBUG] [app] Ending request: urn:request:debcace4-dbe0-4f79-8f7c-216811486b19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:debcace4-dbe0-4f79-8f7c-216811486b19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:37,631 [232] [DEBUG] [app] Starting request: urn:request:190853be-f0f8-49b1-be7b-5c13c3c8bfcd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:54:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:54:37,631 [232] [DEBUG] [app] Ending request: urn:request:190853be-f0f8-49b1-be7b-5c13c3c8bfcd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:190853be-f0f8-49b1-be7b-5c13c3c8bfcd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:54:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( expiredappspecifictokenworker stdout | 2023-01-17 04:54:37,635 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 04:54:37,635 [239] [DEBUG] [app] Starting request: urn:request:01566342-e702-4a0c-adbb-5afd6e6d426a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:37,636 [239] [DEBUG] [app] Ending request: urn:request:01566342-e702-4a0c-adbb-5afd6e6d426a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01566342-e702-4a0c-adbb-5afd6e6d426a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:54:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:37,637 [243] [DEBUG] [app] Starting request: urn:request:0066461a-80a4-4493-ab65-18572d1d9b58 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:37,638 [243] [DEBUG] [app] Ending request: urn:request:0066461a-80a4-4493-ab65-18572d1d9b58 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0066461a-80a4-4493-ab65-18572d1d9b58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:54:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:37,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:37,647 [244] [DEBUG] [app] Ending request: urn:request:14426245-cf15-4233-829e-a9692b0e2241 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14426245-cf15-4233-829e-a9692b0e2241', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:54:37,648 [239] [DEBUG] [app] Ending request: urn:request:44602d30-ceff-4c66-994e-32fa941659ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44602d30-ceff-4c66-994e-32fa941659ab', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:37,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:54:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" blobuploadcleanupworker stdout | 2023-01-17 04:54:37,674 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-registry stdout | 2023-01-17 04:54:37,694 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} securityscanningnotificationworker stdout | 2023-01-17 04:54:37,696 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-registry stdout | 2023-01-17 04:54:37,699 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:54:37,701 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:54:37,705 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:54:37,709 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:54:37,714 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:54:37,717 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:54:37,719 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:54:37,722 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 04:54:38,011 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:54:38,013 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:54:38,015 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:54:38,019 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:54:38,021 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:54:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:43.935629+00:00 (in 5.200094 seconds) notificationworker stdout | 2023-01-17 04:54:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:48 UTC)" (scheduled at 2023-01-17 04:54:38.735066+00:00) notificationworker stdout | 2023-01-17 04:54:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 38, 735964), True, datetime.datetime(2023, 1, 17, 4, 54, 38, 735964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:54:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:54:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:48.738458+00:00 (in 9.707208 seconds) repositorygcworker stdout | 2023-01-17 04:54:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:39 UTC)" (scheduled at 2023-01-17 04:54:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:54:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:54:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 39, 31710), True, datetime.datetime(2023, 1, 17, 4, 54, 39, 31710), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:54:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:54:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:54:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:52.837365+00:00 (in 13.606866 seconds) securityworker stdout | 2023-01-17 04:54:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:09 UTC)" (scheduled at 2023-01-17 04:54:39.230038+00:00) securityworker stdout | 2023-01-17 04:54:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:54:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:54:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:54:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:54:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:54:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:54:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:54:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:48.735066+00:00 (in 4.798906 seconds) notificationworker stdout | 2023-01-17 04:54:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:43 UTC)" (scheduled at 2023-01-17 04:54:43.935629+00:00) notificationworker stdout | 2023-01-17 04:54:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:54:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:43 UTC)" executed successfully builder stdout | 2023-01-17 04:54:45,661 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:45,669 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:45,669 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:45,675 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:54:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:54:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:59.337413+00:00 (in 11.404437 seconds) namespacegcworker stdout | 2023-01-17 04:54:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:47 UTC)" (scheduled at 2023-01-17 04:54:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:54:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:54:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:54:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:58.735066+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 04:54:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:58 UTC)" (scheduled at 2023-01-17 04:54:48.735066+00:00) notificationworker stdout | 2023-01-17 04:54:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 48, 735968), True, datetime.datetime(2023, 1, 17, 4, 54, 48, 735968), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:54:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:54:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:39.030814+00:00 (in 50.291866 seconds) repositorygcworker stdout | 2023-01-17 04:54:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:48 UTC)" (scheduled at 2023-01-17 04:54:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:54:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:54:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:54:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:54:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:54:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:24.641816+00:00 (in 35.811063 seconds) exportactionlogsworker stdout | 2023-01-17 04:54:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:48 UTC)" (scheduled at 2023-01-17 04:54:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:54:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:54:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 48, 831156), True, datetime.datetime(2023, 1, 17, 4, 54, 48, 831156), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:54:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:54:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:54:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:54:52,626 [244] [DEBUG] [app] Starting request: urn:request:3a941752-7852-4e85-9ca4-d1ea54aa2735 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:52,626 [243] [DEBUG] [app] Starting request: urn:request:db1aed4d-8ce9-46e9-8b2f-ddede4a55aaa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:54:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:52,630 [242] [DEBUG] [app] Starting request: urn:request:8659a08c-0024-4423-9153-30a8b911dd49 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:52,631 [236] [DEBUG] [app] Starting request: urn:request:9c872c61-320a-4a33-b81b-e9753c585e8f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:52,631 [242] [DEBUG] [app] Ending request: urn:request:8659a08c-0024-4423-9153-30a8b911dd49 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8659a08c-0024-4423-9153-30a8b911dd49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:54:52,631 [236] [DEBUG] [app] Ending request: urn:request:9c872c61-320a-4a33-b81b-e9753c585e8f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c872c61-320a-4a33-b81b-e9753c585e8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:54:52,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:52,636 [239] [DEBUG] [app] Starting request: urn:request:28c798f8-379c-451a-9755-65dac6e4910d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:52,636 [239] [DEBUG] [app] Ending request: urn:request:28c798f8-379c-451a-9755-65dac6e4910d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28c798f8-379c-451a-9755-65dac6e4910d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:52,636 [241] [DEBUG] [app] Starting request: urn:request:f7ba2253-97ee-4e0f-a1a8-6fc817dffb4d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:52,636 [241] [DEBUG] [app] Ending request: urn:request:f7ba2253-97ee-4e0f-a1a8-6fc817dffb4d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7ba2253-97ee-4e0f-a1a8-6fc817dffb4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:52,648 [244] [DEBUG] [app] Ending request: urn:request:3a941752-7852-4e85-9ca4-d1ea54aa2735 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a941752-7852-4e85-9ca4-d1ea54aa2735', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:52,648 [243] [DEBUG] [app] Ending request: urn:request:db1aed4d-8ce9-46e9-8b2f-ddede4a55aaa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db1aed4d-8ce9-46e9-8b2f-ddede4a55aaa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:54:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:54:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:09.230038+00:00 (in 16.392233 seconds) securityworker stdout | 2023-01-17 04:54:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:22 UTC)" (scheduled at 2023-01-17 04:54:52.837365+00:00) securityworker stdout | 2023-01-17 04:54:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:22 UTC)" executed successfully builder stdout | 2023-01-17 04:54:55,685 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:55,693 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:55,693 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:55,699 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:54:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:54:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:27.226424+00:00 (in 29.999527 seconds) buildlogsarchiver stdout | 2023-01-17 04:54:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:27 UTC)" (scheduled at 2023-01-17 04:54:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:54:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 54, 57, 227278), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:54:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:54:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:54:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:58,488 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:54:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:08.735066+00:00 (in 9.999473 seconds) notificationworker stdout | 2023-01-17 04:54:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:08 UTC)" (scheduled at 2023-01-17 04:54:58.735066+00:00) notificationworker stdout | 2023-01-17 04:54:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 58, 736030), True, datetime.datetime(2023, 1, 17, 4, 54, 58, 736030), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:08 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:54:58,758 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:54:59,001 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:54:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:54:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:47.932490+00:00 (in 48.594639 seconds) namespacegcworker stdout | 2023-01-17 04:54:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:59 UTC)" (scheduled at 2023-01-17 04:54:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:54:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:54:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 59, 338260), True, datetime.datetime(2023, 1, 17, 4, 54, 59, 338260), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:54:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:54:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:54:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:59 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:54:59,796 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:54:59,812 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:55:01,035 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:55:01,845 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:55:01,901 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:55:02,384 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:55:03,117 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:55:03,388 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:55:04,271 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:55:04,790 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:55:05,009 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:55:05,408 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:55:05,411 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:55:05,414 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:55:05,709 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:05,717 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:05,717 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:05,723 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:55:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:55:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:35.848675+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 04:55:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:35 UTC)" (scheduled at 2023-01-17 04:55:05.848675+00:00) gcworker stdout | 2023-01-17 04:55:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721705849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:55:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:55:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:55:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:55:06,172 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:55:06,816 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:55:07,621 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 04:55:07,625 [243] [DEBUG] [app] Starting request: urn:request:fbdce3c3-ed02-4fbe-af18-11af7b6bd8be (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:07,625 [241] [DEBUG] [app] Starting request: urn:request:132b88e6-f5d5-4743-ae38-c92d5a60c017 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:07,630 [232] [DEBUG] [app] Starting request: urn:request:013d1817-9ff2-41b8-9e16-9c9e784c572e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,630 [240] [DEBUG] [app] Starting request: urn:request:d7aded50-d557-469b-abdb-9ef7109f0f32 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,630 [232] [DEBUG] [app] Ending request: urn:request:013d1817-9ff2-41b8-9e16-9c9e784c572e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:013d1817-9ff2-41b8-9e16-9c9e784c572e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:55:07,631 [240] [DEBUG] [app] Ending request: urn:request:d7aded50-d557-469b-abdb-9ef7109f0f32 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d7aded50-d557-469b-abdb-9ef7109f0f32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:55:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:55:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:07,635 [244] [DEBUG] [app] Starting request: urn:request:2522a028-74d2-430b-b088-f08e944bfb53 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:07,635 [244] [DEBUG] [app] Ending request: urn:request:2522a028-74d2-430b-b088-f08e944bfb53 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2522a028-74d2-430b-b088-f08e944bfb53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:07,636 [239] [DEBUG] [app] Starting request: urn:request:d4e712d9-63e0-4ec5-9db7-c4ec28e7fd94 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:07,636 [239] [DEBUG] [app] Ending request: urn:request:d4e712d9-63e0-4ec5-9db7-c4ec28e7fd94 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4e712d9-63e0-4ec5-9db7-c4ec28e7fd94', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:55:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:07,647 [241] [DEBUG] [app] Ending request: urn:request:132b88e6-f5d5-4743-ae38-c92d5a60c017 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:132b88e6-f5d5-4743-ae38-c92d5a60c017', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 04:55:07,651 [243] [DEBUG] [app] Ending request: urn:request:fbdce3c3-ed02-4fbe-af18-11af7b6bd8be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fbdce3c3-ed02-4fbe-af18-11af7b6bd8be', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:07,651 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.027) gunicorn-web stdout | 2023-01-17 04:55:07,651 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" expiredappspecifictokenworker stdout | 2023-01-17 04:55:07,669 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 04:55:07,699 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:55:07,706 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:55:07,708 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} blobuploadcleanupworker stdout | 2023-01-17 04:55:07,710 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-registry stdout | 2023-01-17 04:55:07,713 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:55:07,715 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:55:07,722 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:55:07,725 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:55:07,728 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:55:07,730 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityscanningnotificationworker stdout | 2023-01-17 04:55:07,732 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:55:08,017 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:55:08,019 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:55:08,021 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:55:08,025 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:55:08,028 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:55:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:18.735066+00:00 (in 9.999490 seconds) notificationworker stdout | 2023-01-17 04:55:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:18 UTC)" (scheduled at 2023-01-17 04:55:08.735066+00:00) notificationworker stdout | 2023-01-17 04:55:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 8, 736041), True, datetime.datetime(2023, 1, 17, 4, 55, 8, 736041), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:22.837365+00:00 (in 13.606875 seconds) securityworker stdout | 2023-01-17 04:55:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:39 UTC)" (scheduled at 2023-01-17 04:55:09.230038+00:00) securityworker stdout | 2023-01-17 04:55:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:55:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:55:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:55:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:55:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:55:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:55:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:39 UTC)" executed successfully builder stdout | 2023-01-17 04:55:15,727 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:15,735 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:15,735 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:15,741 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:28.735066+00:00 (in 9.999470 seconds) notificationworker stdout | 2023-01-17 04:55:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:28 UTC)" (scheduled at 2023-01-17 04:55:18.735066+00:00) notificationworker stdout | 2023-01-17 04:55:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 18, 736045), True, datetime.datetime(2023, 1, 17, 4, 55, 18, 736045), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:55:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:55:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:25.334138+00:00 (in 5.091490 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:55:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:20 UTC)" (scheduled at 2023-01-17 04:55:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:55:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:55:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 20, 243083), True, datetime.datetime(2023, 1, 17, 4, 55, 20, 243083), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:55:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:55:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:55:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:55:22,625 [239] [DEBUG] [app] Starting request: urn:request:c3a6d2dc-cf86-4cae-b003-17792e593b71 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:22,625 [241] [DEBUG] [app] Starting request: urn:request:873982ce-cb83-4747-a396-03042a46bd2c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:22,630 [232] [DEBUG] [app] Starting request: urn:request:f6ef7c6f-0ce6-4280-9be6-2544a67b791e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:22,630 [232] [DEBUG] [app] Ending request: urn:request:f6ef7c6f-0ce6-4280-9be6-2544a67b791e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f6ef7c6f-0ce6-4280-9be6-2544a67b791e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:22,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:55:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:55:22,631 [235] [DEBUG] [app] Starting request: urn:request:c5e8ea15-64d4-4047-bf2a-cd86846a842f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:22,632 [235] [DEBUG] [app] Ending request: urn:request:c5e8ea15-64d4-4047-bf2a-cd86846a842f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5e8ea15-64d4-4047-bf2a-cd86846a842f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:55:22,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:22,635 [243] [DEBUG] [app] Starting request: urn:request:3caa4133-95a9-4e4e-b9e0-a3e84cb9b68a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:22,636 [243] [DEBUG] [app] Ending request: urn:request:3caa4133-95a9-4e4e-b9e0-a3e84cb9b68a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3caa4133-95a9-4e4e-b9e0-a3e84cb9b68a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:22,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:22,637 [244] [DEBUG] [app] Starting request: urn:request:820da677-7942-441a-82d0-a974312a10e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:22,637 [244] [DEBUG] [app] Ending request: urn:request:820da677-7942-441a-82d0-a974312a10e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:820da677-7942-441a-82d0-a974312a10e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:22,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:22,648 [241] [DEBUG] [app] Ending request: urn:request:873982ce-cb83-4747-a396-03042a46bd2c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:873982ce-cb83-4747-a396-03042a46bd2c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:22,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:55:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:55:22,648 [239] [DEBUG] [app] Ending request: urn:request:c3a6d2dc-cf86-4cae-b003-17792e593b71 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3a6d2dc-cf86-4cae-b003-17792e593b71', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:22,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 04:55:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:55:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:39.230038+00:00 (in 16.392065 seconds) securityworker stdout | 2023-01-17 04:55:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:52 UTC)" (scheduled at 2023-01-17 04:55:22.837365+00:00) securityworker stdout | 2023-01-17 04:55:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:55:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:55:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:48.830274+00:00 (in 24.187999 seconds) exportactionlogsworker stdout | 2023-01-17 04:55:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:24 UTC)" (scheduled at 2023-01-17 04:55:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:55:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:55:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:55:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:55:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:20.242198+00:00 (in 54.907567 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:55:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:25 UTC)" (scheduled at 2023-01-17 04:55:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:55:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:55:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:25 UTC)" executed successfully builder stdout | 2023-01-17 04:55:25,746 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:25,754 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:25,754 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:25,760 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:55:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:55:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:57.226424+00:00 (in 29.999518 seconds) buildlogsarchiver stdout | 2023-01-17 04:55:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:57 UTC)" (scheduled at 2023-01-17 04:55:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:55:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 55, 27, 227266), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:55:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:55:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:55:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:28,524 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:55:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:38.735066+00:00 (in 9.999470 seconds) notificationworker stdout | 2023-01-17 04:55:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:38 UTC)" (scheduled at 2023-01-17 04:55:28.735066+00:00) notificationworker stdout | 2023-01-17 04:55:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 28, 736039), True, datetime.datetime(2023, 1, 17, 4, 55, 28, 736039), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:38 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:55:28,784 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:55:29,036 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 04:55:29,822 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:55:29,828 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:55:31,069 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:55:31,878 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:55:31,936 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:55:32,420 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:55:33,153 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:55:33,397 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:55:34,309 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:55:34,803 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:55:35,043 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:55:35,416 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:55:35,418 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:55:35,420 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:55:35,760 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:35,769 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:35,769 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:35,775 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:55:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:55:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:05.848675+00:00 (in 29.999451 seconds) gcworker stdout | 2023-01-17 04:55:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:05 UTC)" (scheduled at 2023-01-17 04:55:35.848675+00:00) gcworker stdout | 2023-01-17 04:55:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721735849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:55:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:55:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:55:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:55:36,203 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:55:36,852 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:55:37,626 [244] [DEBUG] [app] Starting request: urn:request:ecf0322b-9310-4c13-a141-9ba1ebb3c9b4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:37,626 [239] [DEBUG] [app] Starting request: urn:request:d85210f0-6a53-4c67-9be3-79b59b7a1176 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:37,631 [232] [DEBUG] [app] Starting request: urn:request:e5217d22-2701-4861-bc65-e8ebce471ba7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:37,631 [242] [DEBUG] [app] Starting request: urn:request:40765d98-04bc-4679-96ed-e693cff94859 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:37,631 [232] [DEBUG] [app] Ending request: urn:request:e5217d22-2701-4861-bc65-e8ebce471ba7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5217d22-2701-4861-bc65-e8ebce471ba7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:55:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:55:37,632 [242] [DEBUG] [app] Ending request: urn:request:40765d98-04bc-4679-96ed-e693cff94859 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40765d98-04bc-4679-96ed-e693cff94859', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:55:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:37,636 [244] [DEBUG] [app] Starting request: urn:request:f95acf71-3e3d-49d1-a12d-66e947757803 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:37,636 [244] [DEBUG] [app] Ending request: urn:request:f95acf71-3e3d-49d1-a12d-66e947757803 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f95acf71-3e3d-49d1-a12d-66e947757803', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:37,638 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:37,639 [243] [DEBUG] [app] Starting request: urn:request:f68b5839-f7e7-48d5-bc8d-a40544396e0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:37,639 [243] [DEBUG] [app] Ending request: urn:request:f68b5839-f7e7-48d5-bc8d-a40544396e0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f68b5839-f7e7-48d5-bc8d-a40544396e0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:37,639 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:55:37,639 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:37,640 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:37,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:37,648 [239] [DEBUG] [app] Ending request: urn:request:d85210f0-6a53-4c67-9be3-79b59b7a1176 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d85210f0-6a53-4c67-9be3-79b59b7a1176', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:55:37,650 [244] [DEBUG] [app] Ending request: urn:request:ecf0322b-9310-4c13-a141-9ba1ebb3c9b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ecf0322b-9310-4c13-a141-9ba1ebb3c9b4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:37,650 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:37,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) gcworker stdout | 2023-01-17 04:55:37,657 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:55:37,704 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 04:55:37,706 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:55:37,712 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:55:37,718 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:55:37,721 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:55:37,724 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:55:37,728 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:55:37,733 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:55:37,736 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:55:37,738 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} blobuploadcleanupworker stdout | 2023-01-17 04:55:37,745 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:55:37,748 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:55:38,023 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:55:38,026 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:55:38,028 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:55:38,031 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:55:38,033 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:55:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:43.935629+00:00 (in 5.200069 seconds) notificationworker stdout | 2023-01-17 04:55:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:48 UTC)" (scheduled at 2023-01-17 04:55:38.735066+00:00) notificationworker stdout | 2023-01-17 04:55:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 38, 735972), True, datetime.datetime(2023, 1, 17, 4, 55, 38, 735972), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:55:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:55:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:48.738458+00:00 (in 9.707223 seconds) repositorygcworker stdout | 2023-01-17 04:55:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:39 UTC)" (scheduled at 2023-01-17 04:55:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:55:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:55:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 39, 31686), True, datetime.datetime(2023, 1, 17, 4, 55, 39, 31686), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:55:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:55:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:55:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:52.837365+00:00 (in 13.606642 seconds) securityworker stdout | 2023-01-17 04:55:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:09 UTC)" (scheduled at 2023-01-17 04:55:39.230038+00:00) securityworker stdout | 2023-01-17 04:55:39,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:55:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:55:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:55:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:55:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:55:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:55:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:55:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:48.735066+00:00 (in 4.798913 seconds) notificationworker stdout | 2023-01-17 04:55:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:43 UTC)" (scheduled at 2023-01-17 04:55:43.935629+00:00) notificationworker stdout | 2023-01-17 04:55:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:55:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:43 UTC)" executed successfully builder stdout | 2023-01-17 04:55:45,784 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:45,792 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:45,792 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:45,798 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:55:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:55:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:59.337413+00:00 (in 11.404446 seconds) namespacegcworker stdout | 2023-01-17 04:55:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:47 UTC)" (scheduled at 2023-01-17 04:55:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:55:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:55:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:55:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:58.735066+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 04:55:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:58 UTC)" (scheduled at 2023-01-17 04:55:48.735066+00:00) notificationworker stdout | 2023-01-17 04:55:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 48, 735945), True, datetime.datetime(2023, 1, 17, 4, 55, 48, 735945), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:55:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:55:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:39.030814+00:00 (in 50.291835 seconds) repositorygcworker stdout | 2023-01-17 04:55:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:48 UTC)" (scheduled at 2023-01-17 04:55:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:55:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:55:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:55:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:55:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:55:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:24.641816+00:00 (in 35.811096 seconds) exportactionlogsworker stdout | 2023-01-17 04:55:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:48 UTC)" (scheduled at 2023-01-17 04:55:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:55:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:55:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 48, 831109), True, datetime.datetime(2023, 1, 17, 4, 55, 48, 831109), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:55:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:55:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:55:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:55:52,625 [244] [DEBUG] [app] Starting request: urn:request:71125380-929c-495b-872b-2b13ac8eddde (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:52,625 [241] [DEBUG] [app] Starting request: urn:request:f56a587a-5719-409e-91af-5634041125ef (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:55:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:52,631 [232] [DEBUG] [app] Starting request: urn:request:361bedbe-d273-473f-9275-b814552a22cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:52,631 [240] [DEBUG] [app] Starting request: urn:request:5e59c064-d4cd-44f7-ba2a-b5eccd675a9f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:52,631 [232] [DEBUG] [app] Ending request: urn:request:361bedbe-d273-473f-9275-b814552a22cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:361bedbe-d273-473f-9275-b814552a22cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:55:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:55:52,632 [240] [DEBUG] [app] Ending request: urn:request:5e59c064-d4cd-44f7-ba2a-b5eccd675a9f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e59c064-d4cd-44f7-ba2a-b5eccd675a9f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:55:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:55:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:52,636 [244] [DEBUG] [app] Starting request: urn:request:c8b05a3e-8fb8-44dd-97c4-47092cdf0755 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:52,636 [239] [DEBUG] [app] Starting request: urn:request:944c9526-e0ec-485f-94fe-96283e5c266e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:52,636 [244] [DEBUG] [app] Ending request: urn:request:c8b05a3e-8fb8-44dd-97c4-47092cdf0755 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8b05a3e-8fb8-44dd-97c4-47092cdf0755', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:52,636 [239] [DEBUG] [app] Ending request: urn:request:944c9526-e0ec-485f-94fe-96283e5c266e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:944c9526-e0ec-485f-94fe-96283e5c266e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:52,647 [244] [DEBUG] [app] Ending request: urn:request:71125380-929c-495b-872b-2b13ac8eddde (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71125380-929c-495b-872b-2b13ac8eddde', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:52,647 [241] [DEBUG] [app] Ending request: urn:request:f56a587a-5719-409e-91af-5634041125ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f56a587a-5719-409e-91af-5634041125ef', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:55:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:55:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:09.230038+00:00 (in 16.392223 seconds) securityworker stdout | 2023-01-17 04:55:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:22 UTC)" (scheduled at 2023-01-17 04:55:52.837365+00:00) securityworker stdout | 2023-01-17 04:55:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:22 UTC)" executed successfully builder stdout | 2023-01-17 04:55:55,802 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:55,810 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:55,810 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:55,816 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:55:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:55:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:27.226424+00:00 (in 29.999565 seconds) buildlogsarchiver stdout | 2023-01-17 04:55:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:27 UTC)" (scheduled at 2023-01-17 04:55:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:55:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 55, 57, 227193), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:55:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:55:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:55:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:58,556 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:55:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:08.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 04:55:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:08 UTC)" (scheduled at 2023-01-17 04:55:58.735066+00:00) notificationworker stdout | 2023-01-17 04:55:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 58, 736061), True, datetime.datetime(2023, 1, 17, 4, 55, 58, 736061), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:08 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:55:58,808 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:55:59,071 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:55:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:55:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:47.932490+00:00 (in 48.594632 seconds) namespacegcworker stdout | 2023-01-17 04:55:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:59 UTC)" (scheduled at 2023-01-17 04:55:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:55:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:55:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 59, 338283), True, datetime.datetime(2023, 1, 17, 4, 55, 59, 338283), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:55:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:55:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:55:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:59 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:55:59,858 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:55:59,863 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:56:01,088 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:56:01,914 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:56:01,961 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:56:02,454 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:56:03,159 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:56:03,405 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:56:04,344 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:56:04,839 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:56:05,078 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:56:05,423 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:56:05,425 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:56:05,428 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 04:56:05,826 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:05,834 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:05,834 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:05,840 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:56:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:56:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:35.848675+00:00 (in 29.999531 seconds) gcworker stdout | 2023-01-17 04:56:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:35 UTC)" (scheduled at 2023-01-17 04:56:05.848675+00:00) gcworker stdout | 2023-01-17 04:56:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721765849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:56:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:56:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:56:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 04:56:06,239 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:56:06,888 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:56:07,626 [239] [DEBUG] [app] Starting request: urn:request:e17cd9cc-fdba-41d8-9287-62800d5e963a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:07,626 [243] [DEBUG] [app] Starting request: urn:request:5d287fa5-1177-4447-9f5b-11e451e6cf46 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:07,631 [240] [DEBUG] [app] Starting request: urn:request:02a1518f-e218-4761-9235-67ab7cc9b832 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,631 [240] [DEBUG] [app] Ending request: urn:request:02a1518f-e218-4761-9235-67ab7cc9b832 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02a1518f-e218-4761-9235-67ab7cc9b832', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,631 [237] [DEBUG] [app] Starting request: urn:request:418bbe76-d49f-4da1-83d8-8a4295cd132b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:56:07,632 [237] [DEBUG] [app] Ending request: urn:request:418bbe76-d49f-4da1-83d8-8a4295cd132b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:418bbe76-d49f-4da1-83d8-8a4295cd132b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:56:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:56:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:07,636 [239] [DEBUG] [app] Starting request: urn:request:0c2256ef-6493-40a5-83cb-38a930f9988f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:07,637 [239] [DEBUG] [app] Ending request: urn:request:0c2256ef-6493-40a5-83cb-38a930f9988f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c2256ef-6493-40a5-83cb-38a930f9988f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:07,637 [244] [DEBUG] [app] Starting request: urn:request:b25dd246-7807-4836-b3e5-85f527d39ff2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:56:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:07,638 [244] [DEBUG] [app] Ending request: urn:request:b25dd246-7807-4836-b3e5-85f527d39ff2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b25dd246-7807-4836-b3e5-85f527d39ff2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:07,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:07,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:56:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:07,648 [239] [DEBUG] [app] Ending request: urn:request:e17cd9cc-fdba-41d8-9287-62800d5e963a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e17cd9cc-fdba-41d8-9287-62800d5e963a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:07,648 [243] [DEBUG] [app] Ending request: urn:request:5d287fa5-1177-4447-9f5b-11e451e6cf46 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d287fa5-1177-4447-9f5b-11e451e6cf46', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:56:07,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:56:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gcworker stdout | 2023-01-17 04:56:07,680 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-registry stdout | 2023-01-17 04:56:07,712 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:56:07,719 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} expiredappspecifictokenworker stdout | 2023-01-17 04:56:07,721 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 04:56:07,724 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:56:07,727 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:56:07,730 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:56:07,735 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:56:07,739 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:56:07,742 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:56:07,744 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} blobuploadcleanupworker stdout | 2023-01-17 04:56:07,754 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:56:07,756 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:56:08,030 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:56:08,033 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:56:08,036 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:56:08,038 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:56:08,040 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:56:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:18.735066+00:00 (in 9.999457 seconds) notificationworker stdout | 2023-01-17 04:56:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:18 UTC)" (scheduled at 2023-01-17 04:56:08.735066+00:00) notificationworker stdout | 2023-01-17 04:56:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 8, 736047), True, datetime.datetime(2023, 1, 17, 4, 56, 8, 736047), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:22.837365+00:00 (in 13.606875 seconds) securityworker stdout | 2023-01-17 04:56:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:39 UTC)" (scheduled at 2023-01-17 04:56:09.230038+00:00) securityworker stdout | 2023-01-17 04:56:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:56:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:56:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:56:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:56:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:56:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:56:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:39 UTC)" executed successfully builder stdout | 2023-01-17 04:56:15,842 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:15,850 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:15,851 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:15,856 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:28.735066+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 04:56:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:28 UTC)" (scheduled at 2023-01-17 04:56:18.735066+00:00) notificationworker stdout | 2023-01-17 04:56:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 18, 735937), True, datetime.datetime(2023, 1, 17, 4, 56, 18, 735937), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:56:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:56:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:25.334138+00:00 (in 5.091393 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:56:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:20 UTC)" (scheduled at 2023-01-17 04:56:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:56:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:56:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 20, 243202), True, datetime.datetime(2023, 1, 17, 4, 56, 20, 243202), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:56:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:56:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:56:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:56:22,625 [239] [DEBUG] [app] Starting request: urn:request:78833e13-4a12-4639-9f38-c2643b924cfe (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:22,625 [244] [DEBUG] [app] Starting request: urn:request:1f665405-7af6-4769-a85a-fee2d6c57f85 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:22,630 [240] [DEBUG] [app] Starting request: urn:request:c9922304-270f-4e5d-8e3f-eb346c5b13a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:22,631 [236] [DEBUG] [app] Starting request: urn:request:319fe7cd-cf80-4b3b-8142-c577b0c2d95d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:22,631 [240] [DEBUG] [app] Ending request: urn:request:c9922304-270f-4e5d-8e3f-eb346c5b13a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c9922304-270f-4e5d-8e3f-eb346c5b13a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:56:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:56:22,631 [236] [DEBUG] [app] Ending request: urn:request:319fe7cd-cf80-4b3b-8142-c577b0c2d95d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:319fe7cd-cf80-4b3b-8142-c577b0c2d95d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:56:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:22,636 [243] [DEBUG] [app] Starting request: urn:request:21f78f07-61db-40e4-a9b6-48f10342feab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:22,636 [241] [DEBUG] [app] Starting request: urn:request:d2ee7bb5-0742-4b1b-98d2-b69259cd45f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:22,636 [243] [DEBUG] [app] Ending request: urn:request:21f78f07-61db-40e4-a9b6-48f10342feab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21f78f07-61db-40e4-a9b6-48f10342feab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:22,637 [241] [DEBUG] [app] Ending request: urn:request:d2ee7bb5-0742-4b1b-98d2-b69259cd45f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d2ee7bb5-0742-4b1b-98d2-b69259cd45f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:56:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:22,648 [244] [DEBUG] [app] Ending request: urn:request:1f665405-7af6-4769-a85a-fee2d6c57f85 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1f665405-7af6-4769-a85a-fee2d6c57f85', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:22,648 [239] [DEBUG] [app] Ending request: urn:request:78833e13-4a12-4639-9f38-c2643b924cfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78833e13-4a12-4639-9f38-c2643b924cfe', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:56:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) securityworker stdout | 2023-01-17 04:56:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:39.230038+00:00 (in 16.392175 seconds) securityworker stdout | 2023-01-17 04:56:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:52 UTC)" (scheduled at 2023-01-17 04:56:22.837365+00:00) securityworker stdout | 2023-01-17 04:56:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:56:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:56:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:44.641497+00:00 (in 19.999201 seconds) exportactionlogsworker stdout | 2023-01-17 04:56:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:24 UTC)" (scheduled at 2023-01-17 04:56:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:56:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:56:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:56:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:56:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:39.333860+00:00 (in 13.999228 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:56:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:25 UTC)" (scheduled at 2023-01-17 04:56:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:56:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:56:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:25 UTC)" executed successfully builder stdout | 2023-01-17 04:56:25,867 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:25,875 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:25,875 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:25,881 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:56:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:56:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:57.226424+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 04:56:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:57 UTC)" (scheduled at 2023-01-17 04:56:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:56:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 56, 27, 227231), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:56:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:56:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:56:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:28,582 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:56:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:38.735066+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 04:56:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:38 UTC)" (scheduled at 2023-01-17 04:56:28.735066+00:00) notificationworker stdout | 2023-01-17 04:56:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 28, 735987), True, datetime.datetime(2023, 1, 17, 4, 56, 28, 735987), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:38 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:56:28,815 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:56:29,106 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 04:56:29,871 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:56:29,893 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:56:31,124 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:56:31,936 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:56:31,995 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:56:32,476 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:56:33,165 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:56:33,420 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:56:34,380 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:56:34,875 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:56:35,096 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:56:35,430 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:56:35,433 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:56:35,435 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:56:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:56:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:05.848675+00:00 (in 29.999539 seconds) gcworker stdout | 2023-01-17 04:56:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:05 UTC)" (scheduled at 2023-01-17 04:56:35.848675+00:00) gcworker stdout | 2023-01-17 04:56:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721795849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:56:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:56:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:56:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:05 UTC)" executed successfully builder stdout | 2023-01-17 04:56:35,883 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:35,891 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:35,891 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:35,896 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:56:36,255 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:56:36,924 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:56:37,625 [244] [DEBUG] [app] Starting request: urn:request:aee4a1f0-be96-49fd-afba-ba4a813ecab2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:37,625 [239] [DEBUG] [app] Starting request: urn:request:e194c9e2-a549-4a40-93c3-803b0b15971d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:37,630 [242] [DEBUG] [app] Starting request: urn:request:8df0a135-4f30-499c-8c6a-9f697b898346 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:37,630 [242] [DEBUG] [app] Ending request: urn:request:8df0a135-4f30-499c-8c6a-9f697b898346 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8df0a135-4f30-499c-8c6a-9f697b898346', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:37,630 [232] [DEBUG] [app] Starting request: urn:request:9479c7c9-4636-46bb-89b9-83193506e75b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:37,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:56:37,631 [232] [DEBUG] [app] Ending request: urn:request:9479c7c9-4636-46bb-89b9-83193506e75b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9479c7c9-4636-46bb-89b9-83193506e75b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:37,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:56:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:56:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:37,634 [241] [DEBUG] [app] Starting request: urn:request:d010c19f-8fa4-4098-a84f-37ef091131cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:37,635 [241] [DEBUG] [app] Ending request: urn:request:d010c19f-8fa4-4098-a84f-37ef091131cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d010c19f-8fa4-4098-a84f-37ef091131cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:37,635 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:37,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:56:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:37,636 [241] [DEBUG] [app] Starting request: urn:request:30e5f913-2c05-4344-9cca-59f04b50ac11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:37,636 [241] [DEBUG] [app] Ending request: urn:request:30e5f913-2c05-4344-9cca-59f04b50ac11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30e5f913-2c05-4344-9cca-59f04b50ac11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:37,640 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:37,646 [239] [DEBUG] [app] Ending request: urn:request:e194c9e2-a549-4a40-93c3-803b0b15971d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e194c9e2-a549-4a40-93c3-803b0b15971d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:37,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:37,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 04:56:37,647 [244] [DEBUG] [app] Ending request: urn:request:aee4a1f0-be96-49fd-afba-ba4a813ecab2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aee4a1f0-be96-49fd-afba-ba4a813ecab2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gcworker stdout | 2023-01-17 04:56:37,709 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-registry stdout | 2023-01-17 04:56:37,718 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:56:37,725 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} expiredappspecifictokenworker stdout | 2023-01-17 04:56:37,730 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 04:56:37,732 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:56:37,736 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:56:37,738 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:56:37,740 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:56:37,745 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:56:37,748 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:56:37,750 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} blobuploadcleanupworker stdout | 2023-01-17 04:56:37,789 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 04:56:37,791 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 04:56:38,037 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:56:38,039 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:56:38,041 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:56:38,044 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:56:38,046 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:56:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:43.935629+00:00 (in 5.200060 seconds) notificationworker stdout | 2023-01-17 04:56:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:48 UTC)" (scheduled at 2023-01-17 04:56:38.735066+00:00) notificationworker stdout | 2023-01-17 04:56:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 38, 735965), True, datetime.datetime(2023, 1, 17, 4, 56, 38, 735965), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:56:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:56:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:48.738458+00:00 (in 9.707133 seconds) repositorygcworker stdout | 2023-01-17 04:56:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:39 UTC)" (scheduled at 2023-01-17 04:56:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:56:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:56:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 39, 31766), True, datetime.datetime(2023, 1, 17, 4, 56, 39, 31766), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:56:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:56:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:56:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:52.837365+00:00 (in 13.606776 seconds) securityworker stdout | 2023-01-17 04:56:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:09 UTC)" (scheduled at 2023-01-17 04:56:39.230038+00:00) securityworker stdout | 2023-01-17 04:56:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:56:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:56:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:56:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:56:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:56:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:20.242198+00:00 (in 40.907899 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:56:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:39 UTC)" (scheduled at 2023-01-17 04:56:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:56:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 56, 39, 334685), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:56:39,342 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 39, 334685), True, datetime.datetime(2023, 1, 17, 4, 56, 39, 334685), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:56:39,344 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 39, 334685), True, datetime.datetime(2023, 1, 17, 4, 56, 39, 334685), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 4, 56, 39, 334685), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:56:39,346 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:56:39,346 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:56:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:48.735066+00:00 (in 4.798940 seconds) notificationworker stdout | 2023-01-17 04:56:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:43 UTC)" (scheduled at 2023-01-17 04:56:43.935629+00:00) notificationworker stdout | 2023-01-17 04:56:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:56:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:56:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:56:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:48.830274+00:00 (in 4.188246 seconds) exportactionlogsworker stdout | 2023-01-17 04:56:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:44 UTC)" (scheduled at 2023-01-17 04:56:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 04:56:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 56, 44, 642404), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:56:44,649 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 44, 642404), True, datetime.datetime(2023, 1, 17, 4, 56, 44, 642404), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:56:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 44, 642404), True, datetime.datetime(2023, 1, 17, 4, 56, 44, 642404), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 4, 56, 44, 642404), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:56:44,653 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:56:44,653 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:44 UTC)" executed successfully builder stdout | 2023-01-17 04:56:45,907 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:45,915 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:45,915 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:45,921 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:56:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:56:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:59.337413+00:00 (in 11.404446 seconds) namespacegcworker stdout | 2023-01-17 04:56:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:47 UTC)" (scheduled at 2023-01-17 04:56:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:56:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:56:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:56:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:58.735066+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 04:56:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:58 UTC)" (scheduled at 2023-01-17 04:56:48.735066+00:00) notificationworker stdout | 2023-01-17 04:56:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 48, 735951), True, datetime.datetime(2023, 1, 17, 4, 56, 48, 735951), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:56:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:56:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:04.738062+00:00 (in 15.999133 seconds) repositorygcworker stdout | 2023-01-17 04:56:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:48 UTC)" (scheduled at 2023-01-17 04:56:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:56:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:56:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:56:48,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:48,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:48,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:56:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:56:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:24.641816+00:00 (in 35.811108 seconds) exportactionlogsworker stdout | 2023-01-17 04:56:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:48 UTC)" (scheduled at 2023-01-17 04:56:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:56:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:56:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 48, 831084), True, datetime.datetime(2023, 1, 17, 4, 56, 48, 831084), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:56:48,840 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:56:48,840 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:56:48,840 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:56:52,625 [244] [DEBUG] [app] Starting request: urn:request:cc7f07cf-d416-4fda-b493-d456316dce45 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:52,625 [239] [DEBUG] [app] Starting request: urn:request:4c378ddb-bf55-4104-b62f-a384c30bf00f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:56:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:52,630 [242] [DEBUG] [app] Starting request: urn:request:8d5efd3d-1139-4a94-bb85-f684fb7ad23e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:52,630 [240] [DEBUG] [app] Starting request: urn:request:b2d48ce5-c06e-407b-a357-e0207d33b37f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:52,630 [242] [DEBUG] [app] Ending request: urn:request:8d5efd3d-1139-4a94-bb85-f684fb7ad23e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d5efd3d-1139-4a94-bb85-f684fb7ad23e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:56:52,631 [240] [DEBUG] [app] Ending request: urn:request:b2d48ce5-c06e-407b-a357-e0207d33b37f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2d48ce5-c06e-407b-a357-e0207d33b37f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:56:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:56:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:56:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:52,635 [243] [DEBUG] [app] Starting request: urn:request:900f0333-640e-4afc-b3fd-7789382157ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:52,635 [241] [DEBUG] [app] Starting request: urn:request:86513b82-4eda-4abf-af8b-1343beb403b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:52,636 [243] [DEBUG] [app] Ending request: urn:request:900f0333-640e-4afc-b3fd-7789382157ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:900f0333-640e-4afc-b3fd-7789382157ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:52,636 [241] [DEBUG] [app] Ending request: urn:request:86513b82-4eda-4abf-af8b-1343beb403b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86513b82-4eda-4abf-af8b-1343beb403b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:56:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:52,647 [239] [DEBUG] [app] Ending request: urn:request:4c378ddb-bf55-4104-b62f-a384c30bf00f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c378ddb-bf55-4104-b62f-a384c30bf00f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:56:52,647 [244] [DEBUG] [app] Ending request: urn:request:cc7f07cf-d416-4fda-b493-d456316dce45 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc7f07cf-d416-4fda-b493-d456316dce45', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 04:56:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 04:56:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:09.230038+00:00 (in 16.392213 seconds) securityworker stdout | 2023-01-17 04:56:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:22 UTC)" (scheduled at 2023-01-17 04:56:52.837365+00:00) securityworker stdout | 2023-01-17 04:56:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:22 UTC)" executed successfully builder stdout | 2023-01-17 04:56:55,931 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:55,940 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:55,940 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:55,946 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:56:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:56:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:27.226424+00:00 (in 29.999533 seconds) buildlogsarchiver stdout | 2023-01-17 04:56:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:27 UTC)" (scheduled at 2023-01-17 04:56:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:56:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 56, 57, 227269), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:56:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:56:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:56:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:58,620 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:56:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:08.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 04:56:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:08 UTC)" (scheduled at 2023-01-17 04:56:58.735066+00:00) notificationworker stdout | 2023-01-17 04:56:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 58, 735979), True, datetime.datetime(2023, 1, 17, 4, 56, 58, 735979), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:08 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:56:58,844 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:56:59,141 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:56:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:56:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:47.932490+00:00 (in 48.594609 seconds) namespacegcworker stdout | 2023-01-17 04:56:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:59 UTC)" (scheduled at 2023-01-17 04:56:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:56:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:56:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 59, 338262), True, datetime.datetime(2023, 1, 17, 4, 56, 59, 338262), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:56:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:56:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:56:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:59 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:56:59,907 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 04:56:59,924 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 04:57:01,154 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:57:01,972 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:57:02,031 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:57:02,512 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:57:03,199 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:57:03,455 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:57:04,392 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} repositorygcworker stdout | 2023-01-17 04:57:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:57:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:39.030814+00:00 (in 34.292291 seconds) repositorygcworker stdout | 2023-01-17 04:57:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:02:04 UTC)" (scheduled at 2023-01-17 04:57:04.738062+00:00) repositorygcworker stdout | 2023-01-17 04:57:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 57, 4, 738913), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:57:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 57, 4, 738913), True, datetime.datetime(2023, 1, 17, 4, 57, 4, 738913), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:57:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 57, 4, 738913), True, datetime.datetime(2023, 1, 17, 4, 57, 4, 738913), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 4, 57, 4, 738913), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:57:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:57:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:02:04 UTC)" executed successfully logrotateworker stdout | 2023-01-17 04:57:04,910 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:57:05,131 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:57:05,438 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:57:05,440 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:57:05,442 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:57:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:57:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:35.848675+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 04:57:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:35 UTC)" (scheduled at 2023-01-17 04:57:05.848675+00:00) gcworker stdout | 2023-01-17 04:57:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721825849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:57:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:57:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:57:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:35 UTC)" executed successfully builder stdout | 2023-01-17 04:57:05,946 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:05,955 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:05,955 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:05,960 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:57:06,277 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:57:06,960 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:57:07,625 [243] [DEBUG] [app] Starting request: urn:request:1ead0152-a665-4eae-97a2-e668a6669ba4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:07,625 [239] [DEBUG] [app] Starting request: urn:request:1c1ddfb1-2548-47a5-8fcc-d7013b782922 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:07,630 [240] [DEBUG] [app] Starting request: urn:request:134f243e-064a-4c75-a639-7c15faa8fc7b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:07,630 [235] [DEBUG] [app] Starting request: urn:request:4a5ba07d-c539-4976-b6ac-7ee8ab9819b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:07,630 [240] [DEBUG] [app] Ending request: urn:request:134f243e-064a-4c75-a639-7c15faa8fc7b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:134f243e-064a-4c75-a639-7c15faa8fc7b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:07,631 [235] [DEBUG] [app] Ending request: urn:request:4a5ba07d-c539-4976-b6ac-7ee8ab9819b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a5ba07d-c539-4976-b6ac-7ee8ab9819b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:57:07,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:57:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:07,636 [244] [DEBUG] [app] Starting request: urn:request:0721a546-649e-4479-9504-46e8862e894d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:07,636 [241] [DEBUG] [app] Starting request: urn:request:384d1258-4dc4-4c3b-b945-c6c18a686f12 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:07,636 [244] [DEBUG] [app] Ending request: urn:request:0721a546-649e-4479-9504-46e8862e894d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0721a546-649e-4479-9504-46e8862e894d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:07,636 [241] [DEBUG] [app] Ending request: urn:request:384d1258-4dc4-4c3b-b945-c6c18a686f12 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:384d1258-4dc4-4c3b-b945-c6c18a686f12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:07,648 [243] [DEBUG] [app] Ending request: urn:request:1ead0152-a665-4eae-97a2-e668a6669ba4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ead0152-a665-4eae-97a2-e668a6669ba4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:07,648 [239] [DEBUG] [app] Ending request: urn:request:1c1ddfb1-2548-47a5-8fcc-d7013b782922 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c1ddfb1-2548-47a5-8fcc-d7013b782922', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:57:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-registry stdout | 2023-01-17 04:57:07,725 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:57:07,732 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:57:07,737 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:57:07,741 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gcworker stdout | 2023-01-17 04:57:07,744 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-registry stdout | 2023-01-17 04:57:07,746 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:57:07,748 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:57:07,751 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:57:07,755 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:57:07,757 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} expiredappspecifictokenworker stdout | 2023-01-17 04:57:07,765 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 04:57:07,796 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 04:57:07,815 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:57:08,043 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:57:08,046 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:57:08,048 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:57:08,050 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:57:08,053 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:57:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:18.735066+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 04:57:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:18 UTC)" (scheduled at 2023-01-17 04:57:08.735066+00:00) notificationworker stdout | 2023-01-17 04:57:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 8, 735975), True, datetime.datetime(2023, 1, 17, 4, 57, 8, 735975), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:22.837365+00:00 (in 13.606871 seconds) securityworker stdout | 2023-01-17 04:57:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:39 UTC)" (scheduled at 2023-01-17 04:57:09.230038+00:00) securityworker stdout | 2023-01-17 04:57:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:57:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:57:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:57:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:57:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:57:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:57:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:39 UTC)" executed successfully builder stdout | 2023-01-17 04:57:15,971 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:15,979 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:15,979 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:15,985 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:28.735066+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 04:57:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:28 UTC)" (scheduled at 2023-01-17 04:57:18.735066+00:00) notificationworker stdout | 2023-01-17 04:57:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 18, 735933), True, datetime.datetime(2023, 1, 17, 4, 57, 18, 735933), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:57:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:57:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:25.334138+00:00 (in 5.091478 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:57:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:20 UTC)" (scheduled at 2023-01-17 04:57:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:57:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:57:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 20, 243074), True, datetime.datetime(2023, 1, 17, 4, 57, 20, 243074), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:57:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:57:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:57:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:57:22,625 [244] [DEBUG] [app] Starting request: urn:request:dab93a37-96ec-4e2b-8169-e55ae74ab252 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:22,625 [243] [DEBUG] [app] Starting request: urn:request:f5b368bd-0db2-47b4-8201-aa6ac15d5d18 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:22,630 [232] [DEBUG] [app] Starting request: urn:request:c362fad9-a3a1-40a4-88ff-6b5b54f347b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,630 [240] [DEBUG] [app] Starting request: urn:request:6388b4e4-ec19-4f15-a1dd-d7282789648a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,631 [232] [DEBUG] [app] Ending request: urn:request:c362fad9-a3a1-40a4-88ff-6b5b54f347b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c362fad9-a3a1-40a4-88ff-6b5b54f347b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:57:22,631 [240] [DEBUG] [app] Ending request: urn:request:6388b4e4-ec19-4f15-a1dd-d7282789648a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6388b4e4-ec19-4f15-a1dd-d7282789648a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:57:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:57:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:57:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:22,635 [239] [DEBUG] [app] Starting request: urn:request:83ed451c-7e5c-4c4d-bdff-8cae6c1aa960 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:22,635 [241] [DEBUG] [app] Starting request: urn:request:081857e4-7ade-435f-a1f2-7ce23b1aa33d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:22,636 [239] [DEBUG] [app] Ending request: urn:request:83ed451c-7e5c-4c4d-bdff-8cae6c1aa960 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83ed451c-7e5c-4c4d-bdff-8cae6c1aa960', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:22,636 [241] [DEBUG] [app] Ending request: urn:request:081857e4-7ade-435f-a1f2-7ce23b1aa33d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:081857e4-7ade-435f-a1f2-7ce23b1aa33d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:22,647 [243] [DEBUG] [app] Ending request: urn:request:f5b368bd-0db2-47b4-8201-aa6ac15d5d18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f5b368bd-0db2-47b4-8201-aa6ac15d5d18', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:22,647 [244] [DEBUG] [app] Ending request: urn:request:dab93a37-96ec-4e2b-8169-e55ae74ab252 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dab93a37-96ec-4e2b-8169-e55ae74ab252', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:57:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 04:57:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:39.230038+00:00 (in 16.392197 seconds) securityworker stdout | 2023-01-17 04:57:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:52 UTC)" (scheduled at 2023-01-17 04:57:22.837365+00:00) securityworker stdout | 2023-01-17 04:57:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:57:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:57:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:48.830274+00:00 (in 24.187998 seconds) exportactionlogsworker stdout | 2023-01-17 04:57:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:24 UTC)" (scheduled at 2023-01-17 04:57:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:57:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:57:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:57:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:57:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:20.242198+00:00 (in 54.907593 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:57:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:25 UTC)" (scheduled at 2023-01-17 04:57:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:57:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:57:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:25 UTC)" executed successfully builder stdout | 2023-01-17 04:57:25,995 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:26,004 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:26,004 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:26,009 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:57:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:57:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:57.226424+00:00 (in 29.999555 seconds) buildlogsarchiver stdout | 2023-01-17 04:57:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:57 UTC)" (scheduled at 2023-01-17 04:57:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:57:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 57, 27, 227219), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:57:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:57:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:57:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:28,656 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:57:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:38.735066+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 04:57:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:38 UTC)" (scheduled at 2023-01-17 04:57:28.735066+00:00) notificationworker stdout | 2023-01-17 04:57:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 28, 735996), True, datetime.datetime(2023, 1, 17, 4, 57, 28, 735996), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:38 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:57:28,865 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:57:29,165 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 04:57:29,935 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:57:29,941 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:57:31,190 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:57:31,984 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:57:32,055 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:57:32,547 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:57:33,229 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:57:33,462 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:57:34,423 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:57:34,934 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:57:35,166 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:57:35,444 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:57:35,447 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:57:35,449 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:57:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:57:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:05.848675+00:00 (in 29.999496 seconds) gcworker stdout | 2023-01-17 04:57:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:05 UTC)" (scheduled at 2023-01-17 04:57:35.848675+00:00) gcworker stdout | 2023-01-17 04:57:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721855849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:57:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:57:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:57:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:05 UTC)" executed successfully builder stdout | 2023-01-17 04:57:36,018 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:36,027 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:36,027 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:36,033 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:57:36,312 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:57:36,985 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:57:37,625 [239] [DEBUG] [app] Starting request: urn:request:69f0f86b-1fea-4ca1-8489-d8fd0fa7f07b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:37,625 [243] [DEBUG] [app] Starting request: urn:request:46e93be0-04a2-436e-b01e-5b036c412871 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:37,630 [232] [DEBUG] [app] Starting request: urn:request:cc474f2d-ac73-4e3f-9c32-27a613c03456 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,630 [242] [DEBUG] [app] Starting request: urn:request:865d5447-d416-4028-b285-6bf28a62f8e7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,630 [232] [DEBUG] [app] Ending request: urn:request:cc474f2d-ac73-4e3f-9c32-27a613c03456 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc474f2d-ac73-4e3f-9c32-27a613c03456', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,630 [242] [DEBUG] [app] Ending request: urn:request:865d5447-d416-4028-b285-6bf28a62f8e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:865d5447-d416-4028-b285-6bf28a62f8e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:57:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:57:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:37,635 [244] [DEBUG] [app] Starting request: urn:request:2bab28e3-3e48-409b-a908-362bca5a8ad9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:37,635 [239] [DEBUG] [app] Starting request: urn:request:d8a11995-b4f6-45c3-ad56-f48a3aab35ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:37,636 [244] [DEBUG] [app] Ending request: urn:request:2bab28e3-3e48-409b-a908-362bca5a8ad9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2bab28e3-3e48-409b-a908-362bca5a8ad9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:37,636 [239] [DEBUG] [app] Ending request: urn:request:d8a11995-b4f6-45c3-ad56-f48a3aab35ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8a11995-b4f6-45c3-ad56-f48a3aab35ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:37,647 [243] [DEBUG] [app] Ending request: urn:request:46e93be0-04a2-436e-b01e-5b036c412871 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:46e93be0-04a2-436e-b01e-5b036c412871', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:37,647 [239] [DEBUG] [app] Ending request: urn:request:69f0f86b-1fea-4ca1-8489-d8fd0fa7f07b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69f0f86b-1fea-4ca1-8489-d8fd0fa7f07b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:57:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 04:57:37,732 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:57:37,737 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:57:37,744 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:57:37,747 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:57:37,753 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:57:37,758 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:57:37,760 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:57:37,762 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:57:37,765 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 04:57:37,778 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:57:37,790 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 04:57:37,812 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 04:57:37,851 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:57:38,048 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:57:38,052 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:57:38,055 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:57:38,058 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:57:38,060 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:57:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:43.935629+00:00 (in 5.200065 seconds) notificationworker stdout | 2023-01-17 04:57:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:48 UTC)" (scheduled at 2023-01-17 04:57:38.735066+00:00) notificationworker stdout | 2023-01-17 04:57:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 38, 735945), True, datetime.datetime(2023, 1, 17, 4, 57, 38, 735945), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:57:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:57:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:48.738458+00:00 (in 9.707161 seconds) repositorygcworker stdout | 2023-01-17 04:57:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:39 UTC)" (scheduled at 2023-01-17 04:57:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:57:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:57:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 39, 31743), True, datetime.datetime(2023, 1, 17, 4, 57, 39, 31743), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:57:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:57:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:57:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:52.837365+00:00 (in 13.606893 seconds) securityworker stdout | 2023-01-17 04:57:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:09 UTC)" (scheduled at 2023-01-17 04:57:39.230038+00:00) securityworker stdout | 2023-01-17 04:57:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:57:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:57:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:57:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:57:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:48.735066+00:00 (in 4.798976 seconds) notificationworker stdout | 2023-01-17 04:57:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:43 UTC)" (scheduled at 2023-01-17 04:57:43.935629+00:00) notificationworker stdout | 2023-01-17 04:57:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:57:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:43 UTC)" executed successfully builder stdout | 2023-01-17 04:57:46,036 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:46,044 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:46,044 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:46,050 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:57:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:57:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:59.337413+00:00 (in 11.404445 seconds) namespacegcworker stdout | 2023-01-17 04:57:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:47 UTC)" (scheduled at 2023-01-17 04:57:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:57:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:57:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:57:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:58.735066+00:00 (in 9.999478 seconds) notificationworker stdout | 2023-01-17 04:57:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:58 UTC)" (scheduled at 2023-01-17 04:57:48.735066+00:00) notificationworker stdout | 2023-01-17 04:57:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 48, 736011), True, datetime.datetime(2023, 1, 17, 4, 57, 48, 736011), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:57:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:57:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:39.030814+00:00 (in 50.291880 seconds) repositorygcworker stdout | 2023-01-17 04:57:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:48 UTC)" (scheduled at 2023-01-17 04:57:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:57:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:57:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:57:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:57:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:57:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:24.641816+00:00 (in 35.811037 seconds) exportactionlogsworker stdout | 2023-01-17 04:57:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:48 UTC)" (scheduled at 2023-01-17 04:57:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:57:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:57:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 48, 831261), True, datetime.datetime(2023, 1, 17, 4, 57, 48, 831261), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:57:48,842 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:57:48,842 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:57:48,842 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:57:52,625 [244] [DEBUG] [app] Starting request: urn:request:d6cbafb5-4161-41c2-babd-155e7868664d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:52,625 [239] [DEBUG] [app] Starting request: urn:request:b299fa4c-183e-4fcb-9152-220a37c72652 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:57:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:52,630 [232] [DEBUG] [app] Starting request: urn:request:c65869ff-f7da-48ac-adb2-9d7793d7ac96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:52,630 [237] [DEBUG] [app] Starting request: urn:request:35c1dc4c-5bc3-4868-9781-fee611121f26 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:52,630 [232] [DEBUG] [app] Ending request: urn:request:c65869ff-f7da-48ac-adb2-9d7793d7ac96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c65869ff-f7da-48ac-adb2-9d7793d7ac96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:57:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:57:52,631 [237] [DEBUG] [app] Ending request: urn:request:35c1dc4c-5bc3-4868-9781-fee611121f26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35c1dc4c-5bc3-4868-9781-fee611121f26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:57:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:57:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:52,636 [243] [DEBUG] [app] Starting request: urn:request:ebe0eb38-eaa9-4cc8-b336-7d4dabc8c7f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:52,636 [241] [DEBUG] [app] Starting request: urn:request:fe9c490d-58b8-483f-92f8-c4315461fc2e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:52,636 [243] [DEBUG] [app] Ending request: urn:request:ebe0eb38-eaa9-4cc8-b336-7d4dabc8c7f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ebe0eb38-eaa9-4cc8-b336-7d4dabc8c7f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:52,637 [241] [DEBUG] [app] Ending request: urn:request:fe9c490d-58b8-483f-92f8-c4315461fc2e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe9c490d-58b8-483f-92f8-c4315461fc2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:52,647 [244] [DEBUG] [app] Ending request: urn:request:d6cbafb5-4161-41c2-babd-155e7868664d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d6cbafb5-4161-41c2-babd-155e7868664d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:52,648 [239] [DEBUG] [app] Ending request: urn:request:b299fa4c-183e-4fcb-9152-220a37c72652 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b299fa4c-183e-4fcb-9152-220a37c72652', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 04:57:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 04:57:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:09.230038+00:00 (in 16.392215 seconds) securityworker stdout | 2023-01-17 04:57:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:22 UTC)" (scheduled at 2023-01-17 04:57:52.837365+00:00) securityworker stdout | 2023-01-17 04:57:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:22 UTC)" executed successfully builder stdout | 2023-01-17 04:57:56,060 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:56,068 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:56,068 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:56,074 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:57:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:57:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:27.226424+00:00 (in 29.999580 seconds) buildlogsarchiver stdout | 2023-01-17 04:57:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:27 UTC)" (scheduled at 2023-01-17 04:57:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:57:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 57, 57, 227184), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:57:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:57:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:57:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:58,689 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:57:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:08.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 04:57:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:08 UTC)" (scheduled at 2023-01-17 04:57:58.735066+00:00) notificationworker stdout | 2023-01-17 04:57:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 58, 735940), True, datetime.datetime(2023, 1, 17, 4, 57, 58, 735940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:08 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:57:58,896 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:57:59,199 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:57:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:57:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:47.932490+00:00 (in 48.594627 seconds) namespacegcworker stdout | 2023-01-17 04:57:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:59 UTC)" (scheduled at 2023-01-17 04:57:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:57:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:57:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 59, 338343), True, datetime.datetime(2023, 1, 17, 4, 57, 59, 338343), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:57:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:57:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:57:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:59 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:57:59,970 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:57:59,976 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:58:01,226 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:58:02,020 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:58:02,090 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:58:02,584 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:58:03,265 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:58:03,472 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:58:04,435 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:58:04,950 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:58:05,201 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:58:05,451 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:58:05,454 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:58:05,456 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:58:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:58:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:35.848675+00:00 (in 29.999469 seconds) gcworker stdout | 2023-01-17 04:58:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:35 UTC)" (scheduled at 2023-01-17 04:58:05.848675+00:00) gcworker stdout | 2023-01-17 04:58:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721885849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:58:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:58:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:58:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:35 UTC)" executed successfully builder stdout | 2023-01-17 04:58:06,084 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:06,095 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:06,095 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:06,100 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:58:06,348 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:58:06,997 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:58:07,626 [244] [DEBUG] [app] Starting request: urn:request:8e0d5920-a15a-4486-bafc-4deed274c72a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:07,626 [243] [DEBUG] [app] Starting request: urn:request:7e1c5bc9-d96b-494d-8bca-e8bef996e9f4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:07,631 [232] [DEBUG] [app] Starting request: urn:request:7f57506a-7d4d-4770-90cf-c74a058e68d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:07,631 [237] [DEBUG] [app] Starting request: urn:request:7ac472ca-c9fa-4859-931f-a46140929a04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:07,631 [232] [DEBUG] [app] Ending request: urn:request:7f57506a-7d4d-4770-90cf-c74a058e68d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f57506a-7d4d-4770-90cf-c74a058e68d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:58:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:58:07,631 [237] [DEBUG] [app] Ending request: urn:request:7ac472ca-c9fa-4859-931f-a46140929a04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ac472ca-c9fa-4859-931f-a46140929a04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:58:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:58:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:07,636 [241] [DEBUG] [app] Starting request: urn:request:fda8f53e-8f1d-45a2-ba71-c6628f42f93b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:07,636 [239] [DEBUG] [app] Starting request: urn:request:05f10a49-761c-4080-b812-1941f754df2b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:07,636 [241] [DEBUG] [app] Ending request: urn:request:fda8f53e-8f1d-45a2-ba71-c6628f42f93b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fda8f53e-8f1d-45a2-ba71-c6628f42f93b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:07,636 [239] [DEBUG] [app] Ending request: urn:request:05f10a49-761c-4080-b812-1941f754df2b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05f10a49-761c-4080-b812-1941f754df2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:58:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:07,647 [244] [DEBUG] [app] Ending request: urn:request:8e0d5920-a15a-4486-bafc-4deed274c72a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e0d5920-a15a-4486-bafc-4deed274c72a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:07,647 [243] [DEBUG] [app] Ending request: urn:request:7e1c5bc9-d96b-494d-8bca-e8bef996e9f4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e1c5bc9-d96b-494d-8bca-e8bef996e9f4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-registry stdout | 2023-01-17 04:58:07,739 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:58:07,743 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:58:07,750 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:58:07,753 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:58:07,759 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:58:07,764 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:58:07,769 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:58:07,771 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:58:07,774 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 04:58:07,813 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:58:07,826 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 04:58:07,828 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 04:58:07,886 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:58:08,054 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:58:08,059 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:58:08,063 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:58:08,065 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:58:08,067 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:58:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:18.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 04:58:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:18 UTC)" (scheduled at 2023-01-17 04:58:08.735066+00:00) notificationworker stdout | 2023-01-17 04:58:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 8, 735919), True, datetime.datetime(2023, 1, 17, 4, 58, 8, 735919), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:08,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:08,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:08,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:22.837365+00:00 (in 13.606847 seconds) securityworker stdout | 2023-01-17 04:58:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:39 UTC)" (scheduled at 2023-01-17 04:58:09.230038+00:00) securityworker stdout | 2023-01-17 04:58:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:58:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:58:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:09,246 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:09,246 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:58:09,247 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:58:09,247 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:58:09,247 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:09,247 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:09,247 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:09,247 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:58:09,247 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:39 UTC)" executed successfully builder stdout | 2023-01-17 04:58:16,111 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:16,119 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:16,119 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:16,125 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:28.735066+00:00 (in 9.999510 seconds) notificationworker stdout | 2023-01-17 04:58:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:28 UTC)" (scheduled at 2023-01-17 04:58:18.735066+00:00) notificationworker stdout | 2023-01-17 04:58:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 18, 735996), True, datetime.datetime(2023, 1, 17, 4, 58, 18, 735996), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:58:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:58:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:25.334138+00:00 (in 5.091447 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:58:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:20 UTC)" (scheduled at 2023-01-17 04:58:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:58:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:58:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 20, 243194), True, datetime.datetime(2023, 1, 17, 4, 58, 20, 243194), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:58:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:58:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:58:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:58:22,625 [239] [DEBUG] [app] Starting request: urn:request:13975391-417d-4ba5-a745-27b0bd1342fd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,626 [243] [DEBUG] [app] Starting request: urn:request:f2ed0ecf-379d-4856-bc36-79da5578fc8e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:22,631 [240] [DEBUG] [app] Starting request: urn:request:cdb9cba4-4fb4-4045-9531-47b55d8fc141 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:22,631 [232] [DEBUG] [app] Starting request: urn:request:1276478c-b3a8-4cdd-a3ad-2103cd447cd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:22,631 [240] [DEBUG] [app] Ending request: urn:request:cdb9cba4-4fb4-4045-9531-47b55d8fc141 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdb9cba4-4fb4-4045-9531-47b55d8fc141', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:58:22,631 [232] [DEBUG] [app] Ending request: urn:request:1276478c-b3a8-4cdd-a3ad-2103cd447cd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1276478c-b3a8-4cdd-a3ad-2103cd447cd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:58:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:58:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:58:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:22,636 [244] [DEBUG] [app] Starting request: urn:request:85e5a23b-f61f-454f-a85f-9f26a1735002 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,636 [244] [DEBUG] [app] Ending request: urn:request:85e5a23b-f61f-454f-a85f-9f26a1735002 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85e5a23b-f61f-454f-a85f-9f26a1735002', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:22,636 [241] [DEBUG] [app] Starting request: urn:request:3944ae88-9385-45a5-a5fd-be57ff16cccc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:58:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:22,636 [241] [DEBUG] [app] Ending request: urn:request:3944ae88-9385-45a5-a5fd-be57ff16cccc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3944ae88-9385-45a5-a5fd-be57ff16cccc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:22,648 [239] [DEBUG] [app] Ending request: urn:request:13975391-417d-4ba5-a745-27b0bd1342fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13975391-417d-4ba5-a745-27b0bd1342fd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:22,648 [243] [DEBUG] [app] Ending request: urn:request:f2ed0ecf-379d-4856-bc36-79da5578fc8e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f2ed0ecf-379d-4856-bc36-79da5578fc8e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:22,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 04:58:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 04:58:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:39.230038+00:00 (in 16.392200 seconds) securityworker stdout | 2023-01-17 04:58:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:52 UTC)" (scheduled at 2023-01-17 04:58:22.837365+00:00) securityworker stdout | 2023-01-17 04:58:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:58:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:58:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:48.830274+00:00 (in 24.187907 seconds) exportactionlogsworker stdout | 2023-01-17 04:58:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:24 UTC)" (scheduled at 2023-01-17 04:58:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:58:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:58:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:58:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:58:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:20.242198+00:00 (in 54.907475 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:58:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:25 UTC)" (scheduled at 2023-01-17 04:58:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:58:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:58:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:25 UTC)" executed successfully builder stdout | 2023-01-17 04:58:26,135 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:26,144 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:26,144 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:26,150 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:58:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:58:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:57.226424+00:00 (in 29.999561 seconds) buildlogsarchiver stdout | 2023-01-17 04:58:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:57 UTC)" (scheduled at 2023-01-17 04:58:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:58:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 58, 27, 227220), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:58:27,240 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:58:27,240 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:58:27,241 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:57 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:28,720 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:58:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:38.735066+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 04:58:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:38 UTC)" (scheduled at 2023-01-17 04:58:28.735066+00:00) notificationworker stdout | 2023-01-17 04:58:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 28, 736054), True, datetime.datetime(2023, 1, 17, 4, 58, 28, 736054), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:38 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:58:28,932 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:58:29,234 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 04:58:29,977 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:58:30,011 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:58:31,262 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:58:32,056 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:58:32,125 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:58:32,595 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:58:33,300 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:58:33,508 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:58:34,443 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:58:34,986 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:58:35,236 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:58:35,458 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:58:35,461 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:58:35,463 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:58:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:58:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:05.848675+00:00 (in 29.999461 seconds) gcworker stdout | 2023-01-17 04:58:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:05 UTC)" (scheduled at 2023-01-17 04:58:35.848675+00:00) gcworker stdout | 2023-01-17 04:58:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721915849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:58:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:58:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:58:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:05 UTC)" executed successfully builder stdout | 2023-01-17 04:58:36,160 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:36,168 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:36,168 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:36,174 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:58:36,381 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:58:37,030 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:58:37,625 [239] [DEBUG] [app] Starting request: urn:request:4dd97ea8-a379-42f4-8d1a-c3f8bd264c15 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:37,625 [244] [DEBUG] [app] Starting request: urn:request:8c200f92-6879-408f-b166-1402e7d4aca3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:37,630 [232] [DEBUG] [app] Starting request: urn:request:95584891-ff7b-4b9a-8294-81c88a24ea4c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,630 [236] [DEBUG] [app] Starting request: urn:request:d0f71200-0acb-49df-aa17-209802b74226 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,630 [232] [DEBUG] [app] Ending request: urn:request:95584891-ff7b-4b9a-8294-81c88a24ea4c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:95584891-ff7b-4b9a-8294-81c88a24ea4c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 04:58:37,631 [236] [DEBUG] [app] Ending request: urn:request:d0f71200-0acb-49df-aa17-209802b74226 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d0f71200-0acb-49df-aa17-209802b74226', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:58:37,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:37,634 [239] [DEBUG] [app] Starting request: urn:request:06c07eaa-5e99-48b3-993d-a45a87138536 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:37,635 [239] [DEBUG] [app] Ending request: urn:request:06c07eaa-5e99-48b3-993d-a45a87138536 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06c07eaa-5e99-48b3-993d-a45a87138536', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:37,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:37,635 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:37,637 [243] [DEBUG] [app] Starting request: urn:request:8dda4506-6245-4e09-9e6a-52ca781e9ae3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:37,637 [243] [DEBUG] [app] Ending request: urn:request:8dda4506-6245-4e09-9e6a-52ca781e9ae3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8dda4506-6245-4e09-9e6a-52ca781e9ae3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:37,640 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:37,654 [239] [DEBUG] [app] Ending request: urn:request:4dd97ea8-a379-42f4-8d1a-c3f8bd264c15 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4dd97ea8-a379-42f4-8d1a-c3f8bd264c15', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:37,654 [244] [DEBUG] [app] Ending request: urn:request:8c200f92-6879-408f-b166-1402e7d4aca3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8c200f92-6879-408f-b166-1402e7d4aca3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:37,654 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:37,655 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:37,655 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:37,655 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-registry stdout | 2023-01-17 04:58:37,746 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:58:37,749 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:58:37,756 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:58:37,759 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:58:37,765 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:58:37,771 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:58:37,776 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:58:37,780 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:58:37,784 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 04:58:37,828 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 04:58:37,833 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 04:58:37,842 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:58:37,907 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:58:38,060 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:58:38,064 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:58:38,068 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:58:38,071 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:58:38,073 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:58:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:43.935629+00:00 (in 5.200029 seconds) notificationworker stdout | 2023-01-17 04:58:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:48 UTC)" (scheduled at 2023-01-17 04:58:38.735066+00:00) notificationworker stdout | 2023-01-17 04:58:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 38, 735982), True, datetime.datetime(2023, 1, 17, 4, 58, 38, 735982), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:58:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:58:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:48.738458+00:00 (in 9.707212 seconds) repositorygcworker stdout | 2023-01-17 04:58:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:39 UTC)" (scheduled at 2023-01-17 04:58:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:58:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:58:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 39, 31715), True, datetime.datetime(2023, 1, 17, 4, 58, 39, 31715), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:58:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:58:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:58:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:52.837365+00:00 (in 13.606840 seconds) securityworker stdout | 2023-01-17 04:58:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:09 UTC)" (scheduled at 2023-01-17 04:58:39.230038+00:00) securityworker stdout | 2023-01-17 04:58:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:58:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:58:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:58:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:58:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:48.735066+00:00 (in 4.798969 seconds) notificationworker stdout | 2023-01-17 04:58:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:43 UTC)" (scheduled at 2023-01-17 04:58:43.935629+00:00) notificationworker stdout | 2023-01-17 04:58:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:58:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:43 UTC)" executed successfully builder stdout | 2023-01-17 04:58:46,184 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:46,193 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:46,193 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:46,199 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:58:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:58:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:59.337413+00:00 (in 11.404432 seconds) namespacegcworker stdout | 2023-01-17 04:58:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:47 UTC)" (scheduled at 2023-01-17 04:58:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:58:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:58:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:58:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:56.935055+00:00 (in 8.199481 seconds) notificationworker stdout | 2023-01-17 04:58:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:58 UTC)" (scheduled at 2023-01-17 04:58:48.735066+00:00) notificationworker stdout | 2023-01-17 04:58:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 48, 736017), True, datetime.datetime(2023, 1, 17, 4, 58, 48, 736017), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:58:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:58:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:39.030814+00:00 (in 50.291886 seconds) repositorygcworker stdout | 2023-01-17 04:58:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:48 UTC)" (scheduled at 2023-01-17 04:58:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:58:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:58:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:58:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:58:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:58:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:24.641816+00:00 (in 35.811078 seconds) exportactionlogsworker stdout | 2023-01-17 04:58:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:48 UTC)" (scheduled at 2023-01-17 04:58:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:58:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:58:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 48, 831128), True, datetime.datetime(2023, 1, 17, 4, 58, 48, 831128), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:58:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:58:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:58:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:58:52,625 [244] [DEBUG] [app] Starting request: urn:request:59bf7b50-2eda-4552-ab58-773b8dfe5db2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:52,625 [243] [DEBUG] [app] Starting request: urn:request:d2a37c25-5c62-4f6d-894d-3203ca534b21 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:58:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:52,630 [240] [DEBUG] [app] Starting request: urn:request:b2fa3070-800c-440d-b152-2414c52ba2c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,630 [237] [DEBUG] [app] Starting request: urn:request:e2cc1565-ebd7-4189-ab1c-9eae3e96dc83 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,631 [240] [DEBUG] [app] Ending request: urn:request:b2fa3070-800c-440d-b152-2414c52ba2c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2fa3070-800c-440d-b152-2414c52ba2c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,631 [237] [DEBUG] [app] Ending request: urn:request:e2cc1565-ebd7-4189-ab1c-9eae3e96dc83 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e2cc1565-ebd7-4189-ab1c-9eae3e96dc83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:58:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:58:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:52,635 [241] [DEBUG] [app] Starting request: urn:request:7a648ee9-572a-4479-8c48-116abb5d6e84 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:52,635 [243] [DEBUG] [app] Starting request: urn:request:faab6ff7-cf9f-4215-9dd1-c726cf52f9a9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:52,636 [243] [DEBUG] [app] Ending request: urn:request:faab6ff7-cf9f-4215-9dd1-c726cf52f9a9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:faab6ff7-cf9f-4215-9dd1-c726cf52f9a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:52,636 [241] [DEBUG] [app] Ending request: urn:request:7a648ee9-572a-4479-8c48-116abb5d6e84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a648ee9-572a-4479-8c48-116abb5d6e84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:52,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:52,647 [244] [DEBUG] [app] Ending request: urn:request:59bf7b50-2eda-4552-ab58-773b8dfe5db2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:59bf7b50-2eda-4552-ab58-773b8dfe5db2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:52,647 [243] [DEBUG] [app] Ending request: urn:request:d2a37c25-5c62-4f6d-894d-3203ca534b21 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d2a37c25-5c62-4f6d-894d-3203ca534b21', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 04:58:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:09.230038+00:00 (in 16.392201 seconds) securityworker stdout | 2023-01-17 04:58:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:22 UTC)" (scheduled at 2023-01-17 04:58:52.837365+00:00) securityworker stdout | 2023-01-17 04:58:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:22 UTC)" executed successfully builder stdout | 2023-01-17 04:58:56,209 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:56,218 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:56,219 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:56,225 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:58.735066+00:00 (in 1.799499 seconds) notificationworker stdout | 2023-01-17 04:58:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:03:56 UTC)" (scheduled at 2023-01-17 04:58:56.935055+00:00) notificationworker stdout | 2023-01-17 04:58:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 58, 56, 935955), 'notification/%']) notificationworker stdout | 2023-01-17 04:58:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 58, 56, 935955), True, datetime.datetime(2023, 1, 17, 4, 58, 56, 935955), 0, 'notification/%']) notificationworker stdout | 2023-01-17 04:58:56,946 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 58, 56, 935955), True, datetime.datetime(2023, 1, 17, 4, 58, 56, 935955), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 4, 58, 56, 935955), 'notification/%']) notificationworker stdout | 2023-01-17 04:58:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:56,948 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:03:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:58:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:58:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:27.226424+00:00 (in 29.999571 seconds) buildlogsarchiver stdout | 2023-01-17 04:58:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:27 UTC)" (scheduled at 2023-01-17 04:58:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:58:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 58, 57, 227240), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:58:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:58:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:58:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:27 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:58,726 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 04:58:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:08.735066+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 04:58:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:08 UTC)" (scheduled at 2023-01-17 04:58:58.735066+00:00) notificationworker stdout | 2023-01-17 04:58:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 58, 735946), True, datetime.datetime(2023, 1, 17, 4, 58, 58, 735946), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:08 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:58:58,968 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:58:59,240 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:58:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:58:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:15.931994+00:00 (in 16.594102 seconds) namespacegcworker stdout | 2023-01-17 04:58:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:59 UTC)" (scheduled at 2023-01-17 04:58:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:58:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:58:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 59, 338272), True, datetime.datetime(2023, 1, 17, 4, 58, 59, 338272), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:58:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:58:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:58:59,345 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:59 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:59:00,012 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:59:00,044 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:59:01,268 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:59:02,092 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:59:02,160 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:59:02,617 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:59:03,317 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:59:03,525 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:59:04,470 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:59:05,021 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:59:05,270 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:59:05,465 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:59:05,467 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:59:05,470 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:59:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:59:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:35.848675+00:00 (in 29.999491 seconds) gcworker stdout | 2023-01-17 04:59:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:35 UTC)" (scheduled at 2023-01-17 04:59:05.848675+00:00) gcworker stdout | 2023-01-17 04:59:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721945849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:59:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:59:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:59:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:35 UTC)" executed successfully builder stdout | 2023-01-17 04:59:06,235 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:06,243 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:06,243 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:06,250 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:59:06,417 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:59:07,066 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:59:07,625 [243] [DEBUG] [app] Starting request: urn:request:807c7365-f798-4d59-844a-8a0a47ae9dc9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:07,626 [241] [DEBUG] [app] Starting request: urn:request:001f7e8b-3ea5-420e-a571-3efd20549d6e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:07,631 [232] [DEBUG] [app] Starting request: urn:request:275a500b-10f0-4603-ab8d-255966f94856 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,631 [242] [DEBUG] [app] Starting request: urn:request:bf90641f-c27e-403d-bef0-afcbd7ed8b1e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,631 [232] [DEBUG] [app] Ending request: urn:request:275a500b-10f0-4603-ab8d-255966f94856 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:275a500b-10f0-4603-ab8d-255966f94856', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,631 [242] [DEBUG] [app] Ending request: urn:request:bf90641f-c27e-403d-bef0-afcbd7ed8b1e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf90641f-c27e-403d-bef0-afcbd7ed8b1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:59:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:59:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:07,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:07,636 [239] [DEBUG] [app] Starting request: urn:request:23569466-e4ac-47b0-98fe-dd91c10dd224 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:07,636 [243] [DEBUG] [app] Starting request: urn:request:845fbcfd-980b-4bcf-b19f-66bbaaa6e9a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:07,637 [239] [DEBUG] [app] Ending request: urn:request:23569466-e4ac-47b0-98fe-dd91c10dd224 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23569466-e4ac-47b0-98fe-dd91c10dd224', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:07,637 [243] [DEBUG] [app] Ending request: urn:request:845fbcfd-980b-4bcf-b19f-66bbaaa6e9a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:845fbcfd-980b-4bcf-b19f-66bbaaa6e9a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:07,648 [243] [DEBUG] [app] Ending request: urn:request:807c7365-f798-4d59-844a-8a0a47ae9dc9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:807c7365-f798-4d59-844a-8a0a47ae9dc9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:07,648 [241] [DEBUG] [app] Ending request: urn:request:001f7e8b-3ea5-420e-a571-3efd20549d6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:001f7e8b-3ea5-420e-a571-3efd20549d6e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:07,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:59:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-registry stdout | 2023-01-17 04:59:07,752 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:59:07,756 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:59:07,761 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:59:07,765 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:59:07,770 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:59:07,776 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:59:07,782 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:59:07,787 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:59:07,791 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 04:59:07,849 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 04:59:07,856 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 04:59:07,858 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:59:07,924 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:59:08,066 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:59:08,073 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:59:08,075 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:59:08,078 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:59:08,080 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:59:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:18.735066+00:00 (in 9.999564 seconds) notificationworker stdout | 2023-01-17 04:59:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:18 UTC)" (scheduled at 2023-01-17 04:59:08.735066+00:00) notificationworker stdout | 2023-01-17 04:59:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 8, 735887), True, datetime.datetime(2023, 1, 17, 4, 59, 8, 735887), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:18 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:22.837365+00:00 (in 13.606863 seconds) securityworker stdout | 2023-01-17 04:59:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:39 UTC)" (scheduled at 2023-01-17 04:59:09.230038+00:00) securityworker stdout | 2023-01-17 04:59:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:59:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:59:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:59:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:39 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:59:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:59:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:47.932490+00:00 (in 31.999991 seconds) namespacegcworker stdout | 2023-01-17 04:59:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:04:15 UTC)" (scheduled at 2023-01-17 04:59:15.931994+00:00) namespacegcworker stdout | 2023-01-17 04:59:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 59, 15, 932917), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:59:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 59, 15, 932917), True, datetime.datetime(2023, 1, 17, 4, 59, 15, 932917), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:59:15,943 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 59, 15, 932917), True, datetime.datetime(2023, 1, 17, 4, 59, 15, 932917), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 4, 59, 15, 932917), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:59:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:59:15,945 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:04:15 UTC)" executed successfully builder stdout | 2023-01-17 04:59:16,260 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:16,268 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:16,268 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:16,274 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:28.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 04:59:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:28 UTC)" (scheduled at 2023-01-17 04:59:18.735066+00:00) notificationworker stdout | 2023-01-17 04:59:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 18, 736014), True, datetime.datetime(2023, 1, 17, 4, 59, 18, 736014), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:59:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:59:20,243 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:25.334138+00:00 (in 5.091348 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:59:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:20 UTC)" (scheduled at 2023-01-17 04:59:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:59:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:59:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 20, 243291), True, datetime.datetime(2023, 1, 17, 4, 59, 20, 243291), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:59:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:59:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:59:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:59:22,625 [243] [DEBUG] [app] Starting request: urn:request:1288241a-fe25-42b9-b71b-9afb342884c9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:22,625 [239] [DEBUG] [app] Starting request: urn:request:c412f454-a867-41a3-8e36-ad2d945f1ad4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:22,630 [232] [DEBUG] [app] Starting request: urn:request:2852daa0-258b-408c-bb00-55218278bd81 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,630 [237] [DEBUG] [app] Starting request: urn:request:9df602e7-9b12-4eec-8599-7d736a319288 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,631 [232] [DEBUG] [app] Ending request: urn:request:2852daa0-258b-408c-bb00-55218278bd81 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2852daa0-258b-408c-bb00-55218278bd81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:59:22,631 [237] [DEBUG] [app] Ending request: urn:request:9df602e7-9b12-4eec-8599-7d736a319288 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9df602e7-9b12-4eec-8599-7d736a319288', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:59:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:59:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:22,636 [244] [DEBUG] [app] Starting request: urn:request:890e1128-345a-4027-b52a-2be4fe8cd270 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:22,636 [243] [DEBUG] [app] Starting request: urn:request:7833c003-8cb7-4c8f-9614-ed47720fc40f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:22,636 [244] [DEBUG] [app] Ending request: urn:request:890e1128-345a-4027-b52a-2be4fe8cd270 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:890e1128-345a-4027-b52a-2be4fe8cd270', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:22,636 [243] [DEBUG] [app] Ending request: urn:request:7833c003-8cb7-4c8f-9614-ed47720fc40f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7833c003-8cb7-4c8f-9614-ed47720fc40f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:22,647 [239] [DEBUG] [app] Ending request: urn:request:c412f454-a867-41a3-8e36-ad2d945f1ad4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c412f454-a867-41a3-8e36-ad2d945f1ad4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:22,648 [243] [DEBUG] [app] Ending request: urn:request:1288241a-fe25-42b9-b71b-9afb342884c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1288241a-fe25-42b9-b71b-9afb342884c9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:59:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 04:59:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:39.230038+00:00 (in 16.392215 seconds) securityworker stdout | 2023-01-17 04:59:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:52 UTC)" (scheduled at 2023-01-17 04:59:22.837365+00:00) securityworker stdout | 2023-01-17 04:59:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:59:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:59:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:48.830274+00:00 (in 24.187979 seconds) exportactionlogsworker stdout | 2023-01-17 04:59:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:24 UTC)" (scheduled at 2023-01-17 04:59:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 04:59:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:59:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:59:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:59:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:20.242198+00:00 (in 54.907475 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:59:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:25 UTC)" (scheduled at 2023-01-17 04:59:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:59:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:59:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:25 UTC)" executed successfully builder stdout | 2023-01-17 04:59:26,284 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:26,293 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:26,293 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:26,299 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:59:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:59:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:57.226424+00:00 (in 29.999561 seconds) buildlogsarchiver stdout | 2023-01-17 04:59:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:57 UTC)" (scheduled at 2023-01-17 04:59:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:59:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 59, 27, 227199), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:59:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:59:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:59:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:59:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:38.735066+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 04:59:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:38 UTC)" (scheduled at 2023-01-17 04:59:28.735066+00:00) notificationworker stdout | 2023-01-17 04:59:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 28, 735984), True, datetime.datetime(2023, 1, 17, 4, 59, 28, 735984), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:38 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:28,761 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:59:29,004 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:59:29,275 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 04:59:30,045 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 04:59:30,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 04:59:31,275 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 04:59:32,128 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:59:32,196 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:59:32,629 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 04:59:33,354 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 04:59:33,534 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 04:59:34,484 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 04:59:35,057 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 04:59:35,305 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 04:59:35,471 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 04:59:35,473 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:59:35,476 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 04:59:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:59:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:05.848675+00:00 (in 29.999500 seconds) gcworker stdout | 2023-01-17 04:59:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:05 UTC)" (scheduled at 2023-01-17 04:59:35.848675+00:00) gcworker stdout | 2023-01-17 04:59:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721975849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:59:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:59:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:59:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:05 UTC)" executed successfully builder stdout | 2023-01-17 04:59:36,309 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:36,318 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:36,318 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:36,324 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 04:59:36,443 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 04:59:37,086 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 04:59:37,625 [239] [DEBUG] [app] Starting request: urn:request:8f189260-0d5e-42c8-bb12-4347c64507ff (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:37,625 [244] [DEBUG] [app] Starting request: urn:request:d7d280d1-b2d4-40e6-8e31-0e04229bee75 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:37,630 [232] [DEBUG] [app] Starting request: urn:request:4c6db0f0-fc1c-476a-ab7f-02be443989f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:37,631 [240] [DEBUG] [app] Starting request: urn:request:005789c7-9275-4f2b-a19b-0314aa9f1c48 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:37,631 [232] [DEBUG] [app] Ending request: urn:request:4c6db0f0-fc1c-476a-ab7f-02be443989f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c6db0f0-fc1c-476a-ab7f-02be443989f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:59:37,631 [240] [DEBUG] [app] Ending request: urn:request:005789c7-9275-4f2b-a19b-0314aa9f1c48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:005789c7-9275-4f2b-a19b-0314aa9f1c48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:59:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:59:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:37,635 [243] [DEBUG] [app] Starting request: urn:request:87e57aa8-0fc8-4de8-9c31-b0fac096c2b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:37,636 [244] [DEBUG] [app] Starting request: urn:request:820d71c4-daea-4041-aa55-b252f80426b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:37,636 [243] [DEBUG] [app] Ending request: urn:request:87e57aa8-0fc8-4de8-9c31-b0fac096c2b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87e57aa8-0fc8-4de8-9c31-b0fac096c2b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:37,636 [244] [DEBUG] [app] Ending request: urn:request:820d71c4-daea-4041-aa55-b252f80426b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:820d71c4-daea-4041-aa55-b252f80426b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:59:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:37,647 [239] [DEBUG] [app] Ending request: urn:request:8f189260-0d5e-42c8-bb12-4347c64507ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f189260-0d5e-42c8-bb12-4347c64507ff', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:37,647 [244] [DEBUG] [app] Ending request: urn:request:d7d280d1-b2d4-40e6-8e31-0e04229bee75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7d280d1-b2d4-40e6-8e31-0e04229bee75', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:59:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-registry stdout | 2023-01-17 04:59:37,760 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:59:37,763 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:59:37,771 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 04:59:37,773 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:59:37,777 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 04:59:37,783 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 04:59:37,789 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:59:37,794 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:59:37,797 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityscanningnotificationworker stdout | 2023-01-17 04:59:37,861 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gcworker stdout | 2023-01-17 04:59:37,866 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 04:59:37,884 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 04:59:37,959 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:59:38,073 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 04:59:38,079 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 04:59:38,081 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 04:59:38,083 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 04:59:38,086 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:59:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:43.935629+00:00 (in 5.200068 seconds) notificationworker stdout | 2023-01-17 04:59:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:48 UTC)" (scheduled at 2023-01-17 04:59:38.735066+00:00) notificationworker stdout | 2023-01-17 04:59:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 38, 736014), True, datetime.datetime(2023, 1, 17, 4, 59, 38, 736014), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:59:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:59:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:48.738458+00:00 (in 9.707192 seconds) repositorygcworker stdout | 2023-01-17 04:59:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:39 UTC)" (scheduled at 2023-01-17 04:59:39.030814+00:00) repositorygcworker stdout | 2023-01-17 04:59:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:59:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 39, 31753), True, datetime.datetime(2023, 1, 17, 4, 59, 39, 31753), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:59:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:59:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:59:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:39 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:52.837365+00:00 (in 13.606849 seconds) securityworker stdout | 2023-01-17 04:59:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:09 UTC)" (scheduled at 2023-01-17 04:59:39.230038+00:00) securityworker stdout | 2023-01-17 04:59:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:59:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:59:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:59:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:59:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:48.735066+00:00 (in 4.798923 seconds) notificationworker stdout | 2023-01-17 04:59:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:43 UTC)" (scheduled at 2023-01-17 04:59:43.935629+00:00) notificationworker stdout | 2023-01-17 04:59:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:59:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:43 UTC)" executed successfully builder stdout | 2023-01-17 04:59:46,324 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:46,333 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:46,333 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:46,339 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:59:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:59:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:59.337413+00:00 (in 11.404422 seconds) namespacegcworker stdout | 2023-01-17 04:59:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:47 UTC)" (scheduled at 2023-01-17 04:59:47.932490+00:00) namespacegcworker stdout | 2023-01-17 04:59:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:59:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:59:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:58.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 04:59:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:58 UTC)" (scheduled at 2023-01-17 04:59:48.735066+00:00) notificationworker stdout | 2023-01-17 04:59:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 48, 735958), True, datetime.datetime(2023, 1, 17, 4, 59, 48, 735958), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:59:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:59:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:39.030814+00:00 (in 50.291857 seconds) repositorygcworker stdout | 2023-01-17 04:59:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:48 UTC)" (scheduled at 2023-01-17 04:59:48.738458+00:00) repositorygcworker stdout | 2023-01-17 04:59:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:59:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:59:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:59:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:59:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:24.641816+00:00 (in 35.811070 seconds) exportactionlogsworker stdout | 2023-01-17 04:59:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:48 UTC)" (scheduled at 2023-01-17 04:59:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 04:59:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:59:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 48, 831124), True, datetime.datetime(2023, 1, 17, 4, 59, 48, 831124), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:59:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:59:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:59:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:59:52,625 [244] [DEBUG] [app] Starting request: urn:request:354c1882-a14d-454a-94a8-e7eb4c44965e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:52,625 [239] [DEBUG] [app] Starting request: urn:request:cf877ea3-53b8-4e0c-b387-0684fa2cba95 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 04:59:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:52,630 [232] [DEBUG] [app] Starting request: urn:request:a0a2a8d1-5672-4bec-ba11-a7d2281b2cf5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,631 [236] [DEBUG] [app] Starting request: urn:request:b9c60ea1-2c80-465b-b859-7155858c2d6a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,631 [232] [DEBUG] [app] Ending request: urn:request:a0a2a8d1-5672-4bec-ba11-a7d2281b2cf5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0a2a8d1-5672-4bec-ba11-a7d2281b2cf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:59:52,631 [236] [DEBUG] [app] Ending request: urn:request:b9c60ea1-2c80-465b-b859-7155858c2d6a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9c60ea1-2c80-465b-b859-7155858c2d6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:59:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:59:52,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:59:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:52,636 [243] [DEBUG] [app] Starting request: urn:request:f5246756-0abb-4c5f-af5f-d0f68a0b2604 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:52,636 [244] [DEBUG] [app] Starting request: urn:request:5b725e87-6f15-40e4-b29c-f402b6c4ed79 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:52,637 [243] [DEBUG] [app] Ending request: urn:request:f5246756-0abb-4c5f-af5f-d0f68a0b2604 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f5246756-0abb-4c5f-af5f-d0f68a0b2604', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:52,637 [244] [DEBUG] [app] Ending request: urn:request:5b725e87-6f15-40e4-b29c-f402b6c4ed79 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5b725e87-6f15-40e4-b29c-f402b6c4ed79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:59:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:52,648 [239] [DEBUG] [app] Ending request: urn:request:cf877ea3-53b8-4e0c-b387-0684fa2cba95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf877ea3-53b8-4e0c-b387-0684fa2cba95', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:52,648 [244] [DEBUG] [app] Ending request: urn:request:354c1882-a14d-454a-94a8-e7eb4c44965e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:354c1882-a14d-454a-94a8-e7eb4c44965e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:59:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 04:59:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:09.230038+00:00 (in 16.392203 seconds) securityworker stdout | 2023-01-17 04:59:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:22 UTC)" (scheduled at 2023-01-17 04:59:52.837365+00:00) securityworker stdout | 2023-01-17 04:59:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:22 UTC)" executed successfully builder stdout | 2023-01-17 04:59:56,349 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:56,361 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:56,361 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:56,367 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:59:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:59:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:27.226424+00:00 (in 29.999529 seconds) buildlogsarchiver stdout | 2023-01-17 04:59:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:27 UTC)" (scheduled at 2023-01-17 04:59:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 04:59:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 59, 57, 227281), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:59:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:59:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:59:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:59:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:08.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 04:59:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:08 UTC)" (scheduled at 2023-01-17 04:59:58.735066+00:00) notificationworker stdout | 2023-01-17 04:59:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 58, 735966), True, datetime.datetime(2023, 1, 17, 4, 59, 58, 735966), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:08 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:58,782 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:59:59,040 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 04:59:59,299 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 04:59:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:59:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:47.932490+00:00 (in 48.594627 seconds) namespacegcworker stdout | 2023-01-17 04:59:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:59 UTC)" (scheduled at 2023-01-17 04:59:59.337413+00:00) namespacegcworker stdout | 2023-01-17 04:59:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:59:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 59, 338345), True, datetime.datetime(2023, 1, 17, 4, 59, 59, 338345), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:59:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:59:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:59:59,345 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:59 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:00:00,081 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:00:00,102 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:00:01,307 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:00:02,140 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:00:02,216 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:00:02,650 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:00:03,383 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:00:03,569 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:00:04,491 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:00:05,080 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:00:05,338 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:00:05,477 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:00:05,480 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:00:05,482 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:00:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:00:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:35.848675+00:00 (in 29.999455 seconds) gcworker stdout | 2023-01-17 05:00:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:35 UTC)" (scheduled at 2023-01-17 05:00:05.848675+00:00) gcworker stdout | 2023-01-17 05:00:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722005849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:00:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:00:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:00:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:35 UTC)" executed successfully builder stdout | 2023-01-17 05:00:06,378 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:06,387 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:06,387 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:06,392 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:00:06,479 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:00:07,103 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:00:07,625 [243] [DEBUG] [app] Starting request: urn:request:ab5cfd43-f84f-4186-aa7d-960724d80927 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:07,625 [244] [DEBUG] [app] Starting request: urn:request:07c44dd8-95eb-4781-a757-0a7798238dbb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:07,630 [232] [DEBUG] [app] Starting request: urn:request:6d6c03bf-6bbc-418b-a592-10084bd248b5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,630 [240] [DEBUG] [app] Starting request: urn:request:ffe4deef-5a07-4ba2-a43b-a103c8256b76 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,631 [232] [DEBUG] [app] Ending request: urn:request:6d6c03bf-6bbc-418b-a592-10084bd248b5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d6c03bf-6bbc-418b-a592-10084bd248b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,631 [240] [DEBUG] [app] Ending request: urn:request:ffe4deef-5a07-4ba2-a43b-a103c8256b76 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ffe4deef-5a07-4ba2-a43b-a103c8256b76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:00:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:00:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:07,636 [244] [DEBUG] [app] Starting request: urn:request:2348d2fd-87e8-4f54-8d9e-2aee026c0608 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:07,636 [239] [DEBUG] [app] Starting request: urn:request:0a7db4fd-0d68-4c87-9696-47628b820fc4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:07,636 [244] [DEBUG] [app] Ending request: urn:request:2348d2fd-87e8-4f54-8d9e-2aee026c0608 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2348d2fd-87e8-4f54-8d9e-2aee026c0608', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:07,636 [239] [DEBUG] [app] Ending request: urn:request:0a7db4fd-0d68-4c87-9696-47628b820fc4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a7db4fd-0d68-4c87-9696-47628b820fc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:07,647 [243] [DEBUG] [app] Ending request: urn:request:ab5cfd43-f84f-4186-aa7d-960724d80927 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab5cfd43-f84f-4186-aa7d-960724d80927', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:07,647 [244] [DEBUG] [app] Ending request: urn:request:07c44dd8-95eb-4781-a757-0a7798238dbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07c44dd8-95eb-4781-a757-0a7798238dbb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:00:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:00:07,767 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:00:07,770 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:00:07,777 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:00:07,779 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:00:07,783 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:00:07,789 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:00:07,795 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:00:07,802 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:00:07,804 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityscanningnotificationworker stdout | 2023-01-17 05:00:07,894 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gcworker stdout | 2023-01-17 05:00:07,900 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:00:07,917 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:00:07,986 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:00:08,079 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:00:08,085 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:00:08,087 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:00:08,090 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:00:08,092 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:00:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:18.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:00:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:18 UTC)" (scheduled at 2023-01-17 05:00:08.735066+00:00) notificationworker stdout | 2023-01-17 05:00:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 8, 735924), True, datetime.datetime(2023, 1, 17, 5, 0, 8, 735924), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:22.837365+00:00 (in 13.606854 seconds) securityworker stdout | 2023-01-17 05:00:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:39 UTC)" (scheduled at 2023-01-17 05:00:09.230038+00:00) securityworker stdout | 2023-01-17 05:00:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:00:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:00:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:00:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:39 UTC)" executed successfully builder stdout | 2023-01-17 05:00:16,394 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:16,403 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:16,403 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:16,409 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:28.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:00:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:28 UTC)" (scheduled at 2023-01-17 05:00:18.735066+00:00) notificationworker stdout | 2023-01-17 05:00:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 18, 735967), True, datetime.datetime(2023, 1, 17, 5, 0, 18, 735967), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:00:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:00:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:25.334138+00:00 (in 5.091379 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:00:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:20 UTC)" (scheduled at 2023-01-17 05:00:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:00:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:00:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 20, 243237), True, datetime.datetime(2023, 1, 17, 5, 0, 20, 243237), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:00:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:00:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:00:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:00:22,625 [244] [DEBUG] [app] Starting request: urn:request:face9ac1-d7cf-43a6-9812-a5ee0786ce86 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:22,625 [243] [DEBUG] [app] Starting request: urn:request:777c4a6f-c27f-412b-9cf6-c31a71032bc1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:22,630 [240] [DEBUG] [app] Starting request: urn:request:5827c611-e2ab-4ae0-8a2d-b95cd250ede1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,630 [242] [DEBUG] [app] Starting request: urn:request:38ed9aca-3855-483b-b64e-ef5083612dcc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,631 [240] [DEBUG] [app] Ending request: urn:request:5827c611-e2ab-4ae0-8a2d-b95cd250ede1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5827c611-e2ab-4ae0-8a2d-b95cd250ede1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:00:22,631 [242] [DEBUG] [app] Ending request: urn:request:38ed9aca-3855-483b-b64e-ef5083612dcc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:38ed9aca-3855-483b-b64e-ef5083612dcc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:00:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:00:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:22,635 [239] [DEBUG] [app] Starting request: urn:request:c9906c27-7b5b-4b75-ae13-02603caec049 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:22,635 [241] [DEBUG] [app] Starting request: urn:request:fe37f845-50c7-459b-aa63-04d9617ca3f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:22,636 [241] [DEBUG] [app] Ending request: urn:request:fe37f845-50c7-459b-aa63-04d9617ca3f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe37f845-50c7-459b-aa63-04d9617ca3f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:22,636 [239] [DEBUG] [app] Ending request: urn:request:c9906c27-7b5b-4b75-ae13-02603caec049 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9906c27-7b5b-4b75-ae13-02603caec049', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:22,647 [243] [DEBUG] [app] Ending request: urn:request:777c4a6f-c27f-412b-9cf6-c31a71032bc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:777c4a6f-c27f-412b-9cf6-c31a71032bc1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:22,647 [244] [DEBUG] [app] Ending request: urn:request:face9ac1-d7cf-43a6-9812-a5ee0786ce86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:face9ac1-d7cf-43a6-9812-a5ee0786ce86', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:22,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:00:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:00:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:00:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:39.230038+00:00 (in 16.392178 seconds) securityworker stdout | 2023-01-17 05:00:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:52 UTC)" (scheduled at 2023-01-17 05:00:22.837365+00:00) securityworker stdout | 2023-01-17 05:00:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:00:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:00:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:48.830274+00:00 (in 24.187978 seconds) exportactionlogsworker stdout | 2023-01-17 05:00:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:24 UTC)" (scheduled at 2023-01-17 05:00:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:00:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:00:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:00:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:00:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:20.242198+00:00 (in 54.907549 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:00:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:25 UTC)" (scheduled at 2023-01-17 05:00:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:00:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:00:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:25 UTC)" executed successfully builder stdout | 2023-01-17 05:00:26,420 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:26,428 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:26,428 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:26,434 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:00:27,029 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 05:00:27,029 [88] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 05:00:27 UTC)" (scheduled at 2023-01-17 05:00:27.029240+00:00) servicekey stdout | 2023-01-17 05:00:27,030 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:27.029240+00:00 (in 3299.999644 seconds) servicekey stdout | 2023-01-17 05:00:27,030 [88] [DEBUG] [__main__] Starting automatic refresh of service key DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk to new expiration 2023-01-17 07:00:27.030089 servicekey stdout | 2023-01-17 05:00:27,031 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk', datetime.datetime(2023, 1, 10, 5, 0, 27, 30629), None, 1, 0]) servicekey stdout | 2023-01-17 05:00:27,038 [88] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443', 'DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk', 'quay', '{"n": "5tQeMffPOXC_hvyxaQcRQ0dkcGXMzwXNoRyrcogogISxmn6hWRBF7ZZ-OMVUcR_UrZF3D69owat2_XNcA6MYvIknjruzv1F0fBWfkFDLJPUdPtwG3WIPRAzDsukW0eyTXittMoUbSi_Evau-lKOQ1BA7N_EHmXI2CvDBK79zVdKmLE0gC3LLazliiE1K2j1Lpp7GRWGwP2WJoDYMAvaM6tkc0EPX-Gx-hIrfGCAcdZGotCR4ZubY9IPsF5gaLmIj7xbXWvwRdvqU0mWABxDcJs5a6uYWzk1ElGk7ITs-6lghCF_JjAJTQwwpLo1gZe89-zmPvxgpZj9kj6Lx18XqMQ", "e": "AQAB", "kty": "RSA", "kid": "DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk"}', '{"created_by": "CLI tool"}', datetime.datetime(2023, 1, 17, 4, 49, 42, 783113), datetime.datetime(2023, 1, 17, 7, 0, 27, 30089), None, 8, 8]) servicekey stdout | 2023-01-17 05:00:27,042 [88] [DEBUG] [__main__] Finished automatic refresh of service key DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk with new expiration 2023-01-17 07:00:27.030089 servicekey stdout | 2023-01-17 05:00:27,042 [88] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:00:27,042 [88] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 05:55:27 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:00:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:00:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:57.226424+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2023-01-17 05:00:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:57 UTC)" (scheduled at 2023-01-17 05:00:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:00:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 0, 27, 227212), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:00:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:00:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:00:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:38.735066+00:00 (in 9.999492 seconds) notificationworker stdout | 2023-01-17 05:00:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:38 UTC)" (scheduled at 2023-01-17 05:00:28.735066+00:00) notificationworker stdout | 2023-01-17 05:00:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 28, 736009), True, datetime.datetime(2023, 1, 17, 5, 0, 28, 736009), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:28,797 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:00:29,075 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:00:29,334 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:00:30,117 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:00:30,130 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:00:31,330 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:00:32,176 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:00:32,239 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:00:32,686 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:00:33,414 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:00:33,604 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:00:34,527 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:00:35,115 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:00:35,344 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:00:35,484 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:00:35,487 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:00:35,489 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:00:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:00:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:05.848675+00:00 (in 29.999337 seconds) gcworker stdout | 2023-01-17 05:00:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:05 UTC)" (scheduled at 2023-01-17 05:00:35.848675+00:00) gcworker stdout | 2023-01-17 05:00:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722035849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:00:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:00:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:00:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:05 UTC)" executed successfully builder stdout | 2023-01-17 05:00:36,445 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:36,453 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:36,453 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:36,459 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:00:36,501 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:00:37,128 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:00:37,626 [239] [DEBUG] [app] Starting request: urn:request:66e89e3b-c619-408b-bd64-055e464d5b54 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:37,626 [244] [DEBUG] [app] Starting request: urn:request:2ee66ac1-64b2-442e-91ad-149aa0265042 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:37,631 [232] [DEBUG] [app] Starting request: urn:request:c519ffd3-e90b-4d9c-aa44-17f1955c67f0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,631 [242] [DEBUG] [app] Starting request: urn:request:2dc8a082-a4eb-4ad6-9fb6-ba90579b2b52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,632 [232] [DEBUG] [app] Ending request: urn:request:c519ffd3-e90b-4d9c-aa44-17f1955c67f0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c519ffd3-e90b-4d9c-aa44-17f1955c67f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,632 [242] [DEBUG] [app] Ending request: urn:request:2dc8a082-a4eb-4ad6-9fb6-ba90579b2b52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2dc8a082-a4eb-4ad6-9fb6-ba90579b2b52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:00:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:00:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:37,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:37,636 [243] [DEBUG] [app] Starting request: urn:request:8e9a2511-7cda-4688-ae90-fff593144e61 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:37,637 [241] [DEBUG] [app] Starting request: urn:request:4d1f545d-cde4-41d9-83e9-f4a44df16ee0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:37,637 [243] [DEBUG] [app] Ending request: urn:request:8e9a2511-7cda-4688-ae90-fff593144e61 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e9a2511-7cda-4688-ae90-fff593144e61', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:37,637 [241] [DEBUG] [app] Ending request: urn:request:4d1f545d-cde4-41d9-83e9-f4a44df16ee0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d1f545d-cde4-41d9-83e9-f4a44df16ee0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:37,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:00:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:37,648 [239] [DEBUG] [app] Ending request: urn:request:66e89e3b-c619-408b-bd64-055e464d5b54 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66e89e3b-c619-408b-bd64-055e464d5b54', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:37,648 [244] [DEBUG] [app] Ending request: urn:request:2ee66ac1-64b2-442e-91ad-149aa0265042 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ee66ac1-64b2-442e-91ad-149aa0265042', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:37,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:00:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:00:37,775 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:00:37,777 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:00:37,785 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:00:37,787 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:00:37,791 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:00:37,795 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:00:37,801 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:00:37,806 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:00:37,811 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 05:00:37,917 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:00:37,921 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:00:37,929 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:00:38,007 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:00:38,085 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:00:38,090 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:00:38,093 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:00:38,096 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:00:38,098 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:00:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:43.935629+00:00 (in 5.200092 seconds) notificationworker stdout | 2023-01-17 05:00:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:48 UTC)" (scheduled at 2023-01-17 05:00:38.735066+00:00) notificationworker stdout | 2023-01-17 05:00:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 38, 735922), True, datetime.datetime(2023, 1, 17, 5, 0, 38, 735922), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:00:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:00:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:48.738458+00:00 (in 9.707221 seconds) repositorygcworker stdout | 2023-01-17 05:00:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:39 UTC)" (scheduled at 2023-01-17 05:00:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:00:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:00:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 39, 31630), True, datetime.datetime(2023, 1, 17, 5, 0, 39, 31630), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:00:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:00:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:00:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:52.837365+00:00 (in 13.606890 seconds) securityworker stdout | 2023-01-17 05:00:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:09 UTC)" (scheduled at 2023-01-17 05:00:39.230038+00:00) securityworker stdout | 2023-01-17 05:00:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:00:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:00:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:00:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:48.735066+00:00 (in 4.798951 seconds) notificationworker stdout | 2023-01-17 05:00:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:43 UTC)" (scheduled at 2023-01-17 05:00:43.935629+00:00) notificationworker stdout | 2023-01-17 05:00:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:00:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:43 UTC)" executed successfully builder stdout | 2023-01-17 05:00:46,469 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:46,479 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:46,479 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:46,484 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:00:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:00:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:59.337413+00:00 (in 11.404388 seconds) namespacegcworker stdout | 2023-01-17 05:00:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:47 UTC)" (scheduled at 2023-01-17 05:00:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:00:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:00:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:58.735066+00:00 (in 9.999467 seconds) notificationworker stdout | 2023-01-17 05:00:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:58 UTC)" (scheduled at 2023-01-17 05:00:48.735066+00:00) notificationworker stdout | 2023-01-17 05:00:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 48, 736003), True, datetime.datetime(2023, 1, 17, 5, 0, 48, 736003), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:00:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:00:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:39.030814+00:00 (in 50.291891 seconds) repositorygcworker stdout | 2023-01-17 05:00:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:48 UTC)" (scheduled at 2023-01-17 05:00:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:00:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:00:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:00:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:00:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:24.641816+00:00 (in 35.811027 seconds) exportactionlogsworker stdout | 2023-01-17 05:00:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:48 UTC)" (scheduled at 2023-01-17 05:00:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:00:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:00:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 48, 831199), True, datetime.datetime(2023, 1, 17, 5, 0, 48, 831199), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:00:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:00:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:00:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:00:52,625 [243] [DEBUG] [app] Starting request: urn:request:ca908bbb-0421-4047-ae47-c0b4978329fe (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:52,625 [241] [DEBUG] [app] Starting request: urn:request:c09b4611-4020-4787-9c3c-eb9cbd8a2f0b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:00:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:52,630 [242] [DEBUG] [app] Starting request: urn:request:a5dd8fe4-dff1-4a01-8472-cb6a9d266b76 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:52,631 [240] [DEBUG] [app] Starting request: urn:request:2d09ad53-9496-441d-b8f1-d31bcde06334 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:52,631 [242] [DEBUG] [app] Ending request: urn:request:a5dd8fe4-dff1-4a01-8472-cb6a9d266b76 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5dd8fe4-dff1-4a01-8472-cb6a9d266b76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:00:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:00:52,631 [240] [DEBUG] [app] Ending request: urn:request:2d09ad53-9496-441d-b8f1-d31bcde06334 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d09ad53-9496-441d-b8f1-d31bcde06334', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:00:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:52,635 [244] [DEBUG] [app] Starting request: urn:request:91392c77-9f3b-4298-9a1c-516aa1d60632 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:52,636 [244] [DEBUG] [app] Ending request: urn:request:91392c77-9f3b-4298-9a1c-516aa1d60632 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91392c77-9f3b-4298-9a1c-516aa1d60632', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:52,636 [239] [DEBUG] [app] Starting request: urn:request:f4c64a5f-3211-4385-b3c7-243cfc209508 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:52,637 [239] [DEBUG] [app] Ending request: urn:request:f4c64a5f-3211-4385-b3c7-243cfc209508 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4c64a5f-3211-4385-b3c7-243cfc209508', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:52,647 [243] [DEBUG] [app] Ending request: urn:request:ca908bbb-0421-4047-ae47-c0b4978329fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca908bbb-0421-4047-ae47-c0b4978329fe', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:00:52,648 [241] [DEBUG] [app] Ending request: urn:request:c09b4611-4020-4787-9c3c-eb9cbd8a2f0b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c09b4611-4020-4787-9c3c-eb9cbd8a2f0b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:52,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:00:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:00:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:09.230038+00:00 (in 16.392206 seconds) securityworker stdout | 2023-01-17 05:00:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:22 UTC)" (scheduled at 2023-01-17 05:00:52.837365+00:00) securityworker stdout | 2023-01-17 05:00:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:22 UTC)" executed successfully builder stdout | 2023-01-17 05:00:56,495 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:56,503 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:56,504 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:56,509 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:00:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:00:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:27.226424+00:00 (in 29.999552 seconds) buildlogsarchiver stdout | 2023-01-17 05:00:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:27 UTC)" (scheduled at 2023-01-17 05:00:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:00:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 0, 57, 227263), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:00:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:00:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:00:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:08.735066+00:00 (in 9.999499 seconds) notificationworker stdout | 2023-01-17 05:00:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:08 UTC)" (scheduled at 2023-01-17 05:00:58.735066+00:00) notificationworker stdout | 2023-01-17 05:00:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 58, 736034), True, datetime.datetime(2023, 1, 17, 5, 0, 58, 736034), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:58,833 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:00:59,111 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} namespacegcworker stdout | 2023-01-17 05:00:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:00:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:47.932490+00:00 (in 48.594626 seconds) namespacegcworker stdout | 2023-01-17 05:00:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:59 UTC)" (scheduled at 2023-01-17 05:00:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:00:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:00:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 59, 338247), True, datetime.datetime(2023, 1, 17, 5, 0, 59, 338247), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:00:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:00:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:00:59,345 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:59 UTC)" executed successfully servicekey stdout | 2023-01-17 05:00:59,368 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:01:00,131 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:01:00,143 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:01:01,344 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:01:02,185 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:01:02,275 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:01:02,717 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:01:03,449 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:01:03,637 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:01:04,556 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:01:05,151 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:01:05,379 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:01:05,492 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:01:05,494 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:01:05,496 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:01:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:01:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:35.848675+00:00 (in 29.999520 seconds) gcworker stdout | 2023-01-17 05:01:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:35 UTC)" (scheduled at 2023-01-17 05:01:05.848675+00:00) gcworker stdout | 2023-01-17 05:01:05,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:01:05,856 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722065856, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:01:05,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:01:05,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:01:05,860 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:35 UTC)" executed successfully builder stdout | 2023-01-17 05:01:06,515 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:06,525 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:06,525 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics storagereplication stdout | 2023-01-17 05:01:06,530 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:01:06,531 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:07,165 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:01:07,626 [239] [DEBUG] [app] Starting request: urn:request:097b1e62-6a5e-44b4-beb1-8bb04a6de2c3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:07,626 [241] [DEBUG] [app] Starting request: urn:request:bcdd09cd-af4e-48d8-96b6-9bf5f546e45f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:07,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:07,631 [232] [DEBUG] [app] Starting request: urn:request:4f807dbb-7c3e-456c-9fa6-4edf825c08c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:07,632 [242] [DEBUG] [app] Starting request: urn:request:5ebbf388-d4b4-4742-9b69-e46d28819c07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:07,632 [232] [DEBUG] [app] Ending request: urn:request:4f807dbb-7c3e-456c-9fa6-4edf825c08c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f807dbb-7c3e-456c-9fa6-4edf825c08c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:07,632 [242] [DEBUG] [app] Ending request: urn:request:5ebbf388-d4b4-4742-9b69-e46d28819c07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5ebbf388-d4b4-4742-9b69-e46d28819c07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:01:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:01:07,633 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:07,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:07,637 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:07,637 [244] [DEBUG] [app] Starting request: urn:request:9f93c429-d9ed-47b8-bd33-f8a47aff94a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:07,638 [243] [DEBUG] [app] Starting request: urn:request:39973a0f-291e-4015-8746-197c265bed29 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:07,638 [244] [DEBUG] [app] Ending request: urn:request:9f93c429-d9ed-47b8-bd33-f8a47aff94a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f93c429-d9ed-47b8-bd33-f8a47aff94a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:07,638 [243] [DEBUG] [app] Ending request: urn:request:39973a0f-291e-4015-8746-197c265bed29 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39973a0f-291e-4015-8746-197c265bed29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:07,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:07,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:07,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:07,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:07,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:07,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:07,649 [239] [DEBUG] [app] Ending request: urn:request:097b1e62-6a5e-44b4-beb1-8bb04a6de2c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:097b1e62-6a5e-44b4-beb1-8bb04a6de2c3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:07,649 [241] [DEBUG] [app] Ending request: urn:request:bcdd09cd-af4e-48d8-96b6-9bf5f546e45f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bcdd09cd-af4e-48d8-96b6-9bf5f546e45f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:07,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:07,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:01:07,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-registry stdout | 2023-01-17 05:01:07,782 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:01:07,785 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:01:07,795 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:01:07,797 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:01:07,799 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:01:07,802 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:01:07,807 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:01:07,812 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:01:07,819 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 05:01:07,939 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:01:07,956 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:01:07,963 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:01:08,012 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:01:08,091 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:01:08,096 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:01:08,099 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:01:08,102 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:01:08,104 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:01:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:18.735066+00:00 (in 9.999489 seconds) notificationworker stdout | 2023-01-17 05:01:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:18 UTC)" (scheduled at 2023-01-17 05:01:08.735066+00:00) notificationworker stdout | 2023-01-17 05:01:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 8, 735976), True, datetime.datetime(2023, 1, 17, 5, 1, 8, 735976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:22.837365+00:00 (in 13.606844 seconds) securityworker stdout | 2023-01-17 05:01:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:39 UTC)" (scheduled at 2023-01-17 05:01:09.230038+00:00) securityworker stdout | 2023-01-17 05:01:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:01:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:01:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:01:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:39 UTC)" executed successfully builder stdout | 2023-01-17 05:01:16,535 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:16,544 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:16,544 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:16,550 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:28.735066+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 05:01:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:28 UTC)" (scheduled at 2023-01-17 05:01:18.735066+00:00) notificationworker stdout | 2023-01-17 05:01:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 18, 735903), True, datetime.datetime(2023, 1, 17, 5, 1, 18, 735903), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:01:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:01:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:25.334138+00:00 (in 5.091463 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:01:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:20 UTC)" (scheduled at 2023-01-17 05:01:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:01:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:01:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 20, 243125), True, datetime.datetime(2023, 1, 17, 5, 1, 20, 243125), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:01:20,252 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:01:20,252 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:01:20,252 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:01:22,625 [239] [DEBUG] [app] Starting request: urn:request:71efac14-3b3d-4b9f-a22f-235d589367b2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:22,625 [244] [DEBUG] [app] Starting request: urn:request:48dac831-a243-4762-acf5-b161d11e16a6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:22,631 [240] [DEBUG] [app] Starting request: urn:request:2aac5994-56e5-4466-9bf9-7b4330d315ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,631 [237] [DEBUG] [app] Starting request: urn:request:1f0a28fb-ae47-4e6d-80d4-cdea5ec3ce99 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,631 [237] [DEBUG] [app] Ending request: urn:request:1f0a28fb-ae47-4e6d-80d4-cdea5ec3ce99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1f0a28fb-ae47-4e6d-80d4-cdea5ec3ce99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,632 [240] [DEBUG] [app] Ending request: urn:request:2aac5994-56e5-4466-9bf9-7b4330d315ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2aac5994-56e5-4466-9bf9-7b4330d315ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) gunicorn-registry stdout | 2023-01-17 05:01:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:22,636 [243] [DEBUG] [app] Starting request: urn:request:739880f5-fa2b-4209-8430-e6c88a43c97f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:22,636 [244] [DEBUG] [app] Starting request: urn:request:c7cf9f08-7323-4b52-b9fb-0d2aa340ec26 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:22,637 [243] [DEBUG] [app] Ending request: urn:request:739880f5-fa2b-4209-8430-e6c88a43c97f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:739880f5-fa2b-4209-8430-e6c88a43c97f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:22,637 [244] [DEBUG] [app] Ending request: urn:request:c7cf9f08-7323-4b52-b9fb-0d2aa340ec26 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c7cf9f08-7323-4b52-b9fb-0d2aa340ec26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:22,649 [244] [DEBUG] [app] Ending request: urn:request:48dac831-a243-4762-acf5-b161d11e16a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48dac831-a243-4762-acf5-b161d11e16a6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:01:22,650 [239] [DEBUG] [app] Ending request: urn:request:71efac14-3b3d-4b9f-a22f-235d589367b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71efac14-3b3d-4b9f-a22f-235d589367b2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 05:01:22,650 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:22,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) securityworker stdout | 2023-01-17 05:01:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:39.230038+00:00 (in 16.392201 seconds) securityworker stdout | 2023-01-17 05:01:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:52 UTC)" (scheduled at 2023-01-17 05:01:22.837365+00:00) securityworker stdout | 2023-01-17 05:01:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:01:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:01:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:44.641497+00:00 (in 19.999222 seconds) exportactionlogsworker stdout | 2023-01-17 05:01:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:24 UTC)" (scheduled at 2023-01-17 05:01:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:01:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:01:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:01:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:01:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:39.333860+00:00 (in 13.999216 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:01:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:25 UTC)" (scheduled at 2023-01-17 05:01:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:01:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:01:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:25 UTC)" executed successfully builder stdout | 2023-01-17 05:01:26,560 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:26,569 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:26,569 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:26,575 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:01:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:01:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:57.226424+00:00 (in 29.999552 seconds) buildlogsarchiver stdout | 2023-01-17 05:01:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:57 UTC)" (scheduled at 2023-01-17 05:01:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:01:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 1, 27, 227229), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:01:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:01:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:01:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:01:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:38.735066+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:01:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:38 UTC)" (scheduled at 2023-01-17 05:01:28.735066+00:00) notificationworker stdout | 2023-01-17 05:01:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 28, 735934), True, datetime.datetime(2023, 1, 17, 5, 1, 28, 735934), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:28,869 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:01:29,125 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:01:29,388 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:01:30,166 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:01:30,178 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:01:31,365 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:01:32,221 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:01:32,309 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:01:32,752 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:01:33,469 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:01:33,669 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:01:34,570 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:01:35,187 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:01:35,399 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:01:35,497 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:01:35,500 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:01:35,502 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:01:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:01:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:05.848675+00:00 (in 29.999471 seconds) gcworker stdout | 2023-01-17 05:01:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:05 UTC)" (scheduled at 2023-01-17 05:01:35.848675+00:00) gcworker stdout | 2023-01-17 05:01:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722095849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:01:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:01:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:01:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:01:36,537 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:01:36,575 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:36,584 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:36,584 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:36,590 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:37,172 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:01:37,625 [244] [DEBUG] [app] Starting request: urn:request:3040a802-3e1d-4e77-9010-50c35cf0f106 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:37,625 [243] [DEBUG] [app] Starting request: urn:request:b12d5d24-b2ea-4be8-8bd1-beba78fb5707 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:37,630 [232] [DEBUG] [app] Starting request: urn:request:6e5cee10-0a57-44b9-b6ab-dcce3210af66 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,631 [242] [DEBUG] [app] Starting request: urn:request:f6d46dd8-543b-4509-ba20-6f8563226b94 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,631 [232] [DEBUG] [app] Ending request: urn:request:6e5cee10-0a57-44b9-b6ab-dcce3210af66 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e5cee10-0a57-44b9-b6ab-dcce3210af66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,631 [242] [DEBUG] [app] Ending request: urn:request:f6d46dd8-543b-4509-ba20-6f8563226b94 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f6d46dd8-543b-4509-ba20-6f8563226b94', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:01:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:01:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:37,635 [239] [DEBUG] [app] Starting request: urn:request:46ccfa1b-9c6e-4dd7-ab5b-ced709a269ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:37,636 [241] [DEBUG] [app] Starting request: urn:request:1cdddfe9-bd3b-4485-8f9a-9f3c2958e223 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:37,636 [239] [DEBUG] [app] Ending request: urn:request:46ccfa1b-9c6e-4dd7-ab5b-ced709a269ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:46ccfa1b-9c6e-4dd7-ab5b-ced709a269ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:37,636 [241] [DEBUG] [app] Ending request: urn:request:1cdddfe9-bd3b-4485-8f9a-9f3c2958e223 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1cdddfe9-bd3b-4485-8f9a-9f3c2958e223', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:37,647 [243] [DEBUG] [app] Ending request: urn:request:b12d5d24-b2ea-4be8-8bd1-beba78fb5707 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b12d5d24-b2ea-4be8-8bd1-beba78fb5707', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:37,647 [244] [DEBUG] [app] Ending request: urn:request:3040a802-3e1d-4e77-9010-50c35cf0f106 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3040a802-3e1d-4e77-9010-50c35cf0f106', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:37,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:01:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:01:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:01:37,790 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:01:37,792 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:01:37,800 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:01:37,804 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:01:37,806 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:01:37,809 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:01:37,813 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:01:37,822 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:01:37,825 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 05:01:37,944 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:01:37,991 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:01:37,998 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:01:38,020 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:01:38,098 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:01:38,101 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:01:38,105 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:01:38,108 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:01:38,110 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:01:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:43.935629+00:00 (in 5.200049 seconds) notificationworker stdout | 2023-01-17 05:01:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:48 UTC)" (scheduled at 2023-01-17 05:01:38.735066+00:00) notificationworker stdout | 2023-01-17 05:01:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 38, 735998), True, datetime.datetime(2023, 1, 17, 5, 1, 38, 735998), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:01:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:01:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:48.738458+00:00 (in 9.707153 seconds) repositorygcworker stdout | 2023-01-17 05:01:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:39 UTC)" (scheduled at 2023-01-17 05:01:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:01:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:01:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 39, 31831), True, datetime.datetime(2023, 1, 17, 5, 1, 39, 31831), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:01:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:01:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:01:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:52.837365+00:00 (in 13.606865 seconds) securityworker stdout | 2023-01-17 05:01:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:09 UTC)" (scheduled at 2023-01-17 05:01:39.230038+00:00) securityworker stdout | 2023-01-17 05:01:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:01:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:01:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:01:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:01:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:01:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:01:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:01:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:01:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:20.242198+00:00 (in 40.907858 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:01:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:39 UTC)" (scheduled at 2023-01-17 05:01:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:01:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 1, 39, 334747), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:01:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 39, 334747), True, datetime.datetime(2023, 1, 17, 5, 1, 39, 334747), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:01:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 39, 334747), True, datetime.datetime(2023, 1, 17, 5, 1, 39, 334747), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 1, 39, 334747), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:01:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:01:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:01:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:48.735066+00:00 (in 4.798934 seconds) notificationworker stdout | 2023-01-17 05:01:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:43 UTC)" (scheduled at 2023-01-17 05:01:43.935629+00:00) notificationworker stdout | 2023-01-17 05:01:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:01:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:01:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:01:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:48.830274+00:00 (in 4.188267 seconds) exportactionlogsworker stdout | 2023-01-17 05:01:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:44 UTC)" (scheduled at 2023-01-17 05:01:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:01:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 1, 44, 642410), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:01:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 44, 642410), True, datetime.datetime(2023, 1, 17, 5, 1, 44, 642410), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:01:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 44, 642410), True, datetime.datetime(2023, 1, 17, 5, 1, 44, 642410), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 1, 44, 642410), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:01:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:01:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:44 UTC)" executed successfully builder stdout | 2023-01-17 05:01:46,594 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:46,603 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:46,603 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:46,609 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:01:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:01:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:59.337413+00:00 (in 11.404430 seconds) namespacegcworker stdout | 2023-01-17 05:01:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:47 UTC)" (scheduled at 2023-01-17 05:01:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:01:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:01:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:01:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:58.735066+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:01:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:58 UTC)" (scheduled at 2023-01-17 05:01:48.735066+00:00) notificationworker stdout | 2023-01-17 05:01:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 48, 735995), True, datetime.datetime(2023, 1, 17, 5, 1, 48, 735995), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:01:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:01:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:04.738062+00:00 (in 15.999107 seconds) repositorygcworker stdout | 2023-01-17 05:01:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:48 UTC)" (scheduled at 2023-01-17 05:01:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:01:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:01:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:01:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:01:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:01:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:24.641816+00:00 (in 35.811087 seconds) exportactionlogsworker stdout | 2023-01-17 05:01:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:48 UTC)" (scheduled at 2023-01-17 05:01:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:01:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:01:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 48, 831159), True, datetime.datetime(2023, 1, 17, 5, 1, 48, 831159), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:01:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:01:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:01:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:01:52,625 [244] [DEBUG] [app] Starting request: urn:request:d759c620-8239-407f-a10e-8fa37146818a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:52,625 [243] [DEBUG] [app] Starting request: urn:request:233c0eae-e960-4691-ac7d-c396cdd1b6e2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:01:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:52,630 [232] [DEBUG] [app] Starting request: urn:request:9e22d86b-21b7-41a4-bba1-42962bd73d1d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,630 [242] [DEBUG] [app] Starting request: urn:request:8c647ac0-4b66-40d5-b36f-8a64acf0eb49 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,630 [232] [DEBUG] [app] Ending request: urn:request:9e22d86b-21b7-41a4-bba1-42962bd73d1d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e22d86b-21b7-41a4-bba1-42962bd73d1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,631 [242] [DEBUG] [app] Ending request: urn:request:8c647ac0-4b66-40d5-b36f-8a64acf0eb49 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c647ac0-4b66-40d5-b36f-8a64acf0eb49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:01:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:01:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:52,635 [239] [DEBUG] [app] Starting request: urn:request:06aedf5d-1cf0-42be-b142-763a16880565 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:52,636 [239] [DEBUG] [app] Ending request: urn:request:06aedf5d-1cf0-42be-b142-763a16880565 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06aedf5d-1cf0-42be-b142-763a16880565', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:52,636 [243] [DEBUG] [app] Starting request: urn:request:97f00833-585c-44fb-9a37-b62767db7e1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:52,636 [243] [DEBUG] [app] Ending request: urn:request:97f00833-585c-44fb-9a37-b62767db7e1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97f00833-585c-44fb-9a37-b62767db7e1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:52,648 [243] [DEBUG] [app] Ending request: urn:request:233c0eae-e960-4691-ac7d-c396cdd1b6e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:233c0eae-e960-4691-ac7d-c396cdd1b6e2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:52,648 [244] [DEBUG] [app] Ending request: urn:request:d759c620-8239-407f-a10e-8fa37146818a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d759c620-8239-407f-a10e-8fa37146818a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 05:01:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:01:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:09.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 05:01:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:22 UTC)" (scheduled at 2023-01-17 05:01:52.837365+00:00) securityworker stdout | 2023-01-17 05:01:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:22 UTC)" executed successfully builder stdout | 2023-01-17 05:01:56,619 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:56,628 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:56,628 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:56,634 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:01:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:01:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:27.226424+00:00 (in 29.999541 seconds) buildlogsarchiver stdout | 2023-01-17 05:01:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:27 UTC)" (scheduled at 2023-01-17 05:01:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:01:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 1, 57, 227259), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:01:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:01:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:01:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:01:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:08.735066+00:00 (in 9.999491 seconds) notificationworker stdout | 2023-01-17 05:01:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:08 UTC)" (scheduled at 2023-01-17 05:01:58.735066+00:00) notificationworker stdout | 2023-01-17 05:01:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 58, 736016), True, datetime.datetime(2023, 1, 17, 5, 1, 58, 736016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:58,898 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:01:59,161 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} namespacegcworker stdout | 2023-01-17 05:01:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:01:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:47.932490+00:00 (in 48.594634 seconds) namespacegcworker stdout | 2023-01-17 05:01:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:59 UTC)" (scheduled at 2023-01-17 05:01:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:01:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:01:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 59, 338274), True, datetime.datetime(2023, 1, 17, 5, 1, 59, 338274), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:01:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:01:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:01:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:59 UTC)" executed successfully servicekey stdout | 2023-01-17 05:01:59,399 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:02:00,190 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:02:00,193 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:02:01,386 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:02:02,237 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:02:02,343 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:02:02,771 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:02:03,481 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:02:03,676 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:02:04,595 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} repositorygcworker stdout | 2023-01-17 05:02:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:02:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:39.030814+00:00 (in 34.292338 seconds) repositorygcworker stdout | 2023-01-17 05:02:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:07:04 UTC)" (scheduled at 2023-01-17 05:02:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:02:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 2, 4, 738835), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:02:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 2, 4, 738835), True, datetime.datetime(2023, 1, 17, 5, 2, 4, 738835), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:02:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 2, 4, 738835), True, datetime.datetime(2023, 1, 17, 5, 2, 4, 738835), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 2, 4, 738835), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:02:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:02:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:07:04 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:02:05,222 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:02:05,434 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:02:05,505 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:02:05,507 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:02:05,510 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:02:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:02:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:35.848675+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 05:02:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:35 UTC)" (scheduled at 2023-01-17 05:02:05.848675+00:00) gcworker stdout | 2023-01-17 05:02:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722125849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:02:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:02:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:02:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:02:06,573 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:02:06,644 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:06,653 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:06,653 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:06,659 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:07,178 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:02:07,625 [239] [DEBUG] [app] Starting request: urn:request:d1d9a843-7c79-48a2-b49e-52ab36232f04 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:07,625 [241] [DEBUG] [app] Starting request: urn:request:44ad2034-9b2c-44e7-8e23-8de76dd7b639 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:07,631 [232] [DEBUG] [app] Starting request: urn:request:062334a5-7d3b-46f1-99bb-d2ad193b80c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:07,631 [242] [DEBUG] [app] Starting request: urn:request:48375b3a-4302-4862-9577-51e3194f39f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:07,631 [232] [DEBUG] [app] Ending request: urn:request:062334a5-7d3b-46f1-99bb-d2ad193b80c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:062334a5-7d3b-46f1-99bb-d2ad193b80c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:02:07,632 [242] [DEBUG] [app] Ending request: urn:request:48375b3a-4302-4862-9577-51e3194f39f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48375b3a-4302-4862-9577-51e3194f39f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:02:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:07,637 [244] [DEBUG] [app] Starting request: urn:request:a15d9c6b-2b7b-445b-a6db-78ee0a0941e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:07,637 [244] [DEBUG] [app] Ending request: urn:request:a15d9c6b-2b7b-445b-a6db-78ee0a0941e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a15d9c6b-2b7b-445b-a6db-78ee0a0941e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:07,637 [243] [DEBUG] [app] Starting request: urn:request:807db432-9502-4937-b214-acaf03b34863 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:02:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:07,638 [243] [DEBUG] [app] Ending request: urn:request:807db432-9502-4937-b214-acaf03b34863 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:807db432-9502-4937-b214-acaf03b34863', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:07,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:07,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:07,648 [239] [DEBUG] [app] Ending request: urn:request:d1d9a843-7c79-48a2-b49e-52ab36232f04 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1d9a843-7c79-48a2-b49e-52ab36232f04', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:07,648 [241] [DEBUG] [app] Ending request: urn:request:44ad2034-9b2c-44e7-8e23-8de76dd7b639 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44ad2034-9b2c-44e7-8e23-8de76dd7b639', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:07,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:07,649 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:02:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:07,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:02:07,798 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:02:07,800 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:02:07,806 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:02:07,811 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:02:07,814 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:02:07,816 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:02:07,820 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:02:07,831 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:02:07,833 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gcworker stdout | 2023-01-17 05:02:07,971 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:02:08,026 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:02:08,028 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:02:08,050 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:02:08,103 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:02:08,107 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:02:08,110 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:02:08,115 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:02:08,117 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:02:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:18.735066+00:00 (in 9.999525 seconds) notificationworker stdout | 2023-01-17 05:02:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:18 UTC)" (scheduled at 2023-01-17 05:02:08.735066+00:00) notificationworker stdout | 2023-01-17 05:02:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 8, 735976), True, datetime.datetime(2023, 1, 17, 5, 2, 8, 735976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:02:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:22.837365+00:00 (in 13.606851 seconds) securityworker stdout | 2023-01-17 05:02:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:39 UTC)" (scheduled at 2023-01-17 05:02:09.230038+00:00) securityworker stdout | 2023-01-17 05:02:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:02:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:02:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:02:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:39 UTC)" executed successfully builder stdout | 2023-01-17 05:02:16,669 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:16,678 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:16,679 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:16,684 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:28.735066+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:02:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:28 UTC)" (scheduled at 2023-01-17 05:02:18.735066+00:00) notificationworker stdout | 2023-01-17 05:02:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 18, 735931), True, datetime.datetime(2023, 1, 17, 5, 2, 18, 735931), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:02:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:02:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:25.334138+00:00 (in 5.091407 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:02:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:20 UTC)" (scheduled at 2023-01-17 05:02:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:02:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:02:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 20, 243175), True, datetime.datetime(2023, 1, 17, 5, 2, 20, 243175), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:02:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:02:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:02:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:02:22,625 [239] [DEBUG] [app] Starting request: urn:request:d09a6d62-0916-4648-b05a-c5d45ca59d65 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,625 [241] [DEBUG] [app] Starting request: urn:request:a2c4e4d5-2eaf-454b-b83a-c1b515f3e719 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:22,630 [232] [DEBUG] [app] Starting request: urn:request:dc88521e-03b9-4739-bb5b-2c3f2aafd679 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:22,630 [242] [DEBUG] [app] Starting request: urn:request:e3ac9d74-74b7-4560-8eae-20a8f1cff424 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:22,630 [232] [DEBUG] [app] Ending request: urn:request:dc88521e-03b9-4739-bb5b-2c3f2aafd679 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc88521e-03b9-4739-bb5b-2c3f2aafd679', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:02:22,631 [242] [DEBUG] [app] Ending request: urn:request:e3ac9d74-74b7-4560-8eae-20a8f1cff424 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e3ac9d74-74b7-4560-8eae-20a8f1cff424', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:02:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:22,635 [244] [DEBUG] [app] Starting request: urn:request:0830fafa-6e34-4633-b87c-448077bb648c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,636 [244] [DEBUG] [app] Ending request: urn:request:0830fafa-6e34-4633-b87c-448077bb648c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0830fafa-6e34-4633-b87c-448077bb648c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:22,636 [243] [DEBUG] [app] Starting request: urn:request:2d7da803-c6e4-41fc-9a42-9268b13d9ffc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:02:22,636 [243] [DEBUG] [app] Ending request: urn:request:2d7da803-c6e4-41fc-9a42-9268b13d9ffc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2d7da803-c6e4-41fc-9a42-9268b13d9ffc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:22,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:22,647 [239] [DEBUG] [app] Ending request: urn:request:d09a6d62-0916-4648-b05a-c5d45ca59d65 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d09a6d62-0916-4648-b05a-c5d45ca59d65', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:22,647 [241] [DEBUG] [app] Ending request: urn:request:a2c4e4d5-2eaf-454b-b83a-c1b515f3e719 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2c4e4d5-2eaf-454b-b83a-c1b515f3e719', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:22,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:22,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:02:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:39.230038+00:00 (in 16.392184 seconds) securityworker stdout | 2023-01-17 05:02:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:52 UTC)" (scheduled at 2023-01-17 05:02:22.837365+00:00) securityworker stdout | 2023-01-17 05:02:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:02:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:02:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:48.830274+00:00 (in 24.187969 seconds) exportactionlogsworker stdout | 2023-01-17 05:02:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:24 UTC)" (scheduled at 2023-01-17 05:02:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:02:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:02:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:02:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:02:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:20.242198+00:00 (in 54.907570 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:02:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:25 UTC)" (scheduled at 2023-01-17 05:02:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:02:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:02:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:25 UTC)" executed successfully builder stdout | 2023-01-17 05:02:26,694 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:26,703 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:26,703 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:26,709 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:02:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:02:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:57.226424+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:02:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:57 UTC)" (scheduled at 2023-01-17 05:02:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:02:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 2, 27, 227252), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:02:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:02:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:02:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:38.735066+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 05:02:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:38 UTC)" (scheduled at 2023-01-17 05:02:28.735066+00:00) notificationworker stdout | 2023-01-17 05:02:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 28, 735985), True, datetime.datetime(2023, 1, 17, 5, 2, 28, 735985), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:02:28,920 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:02:29,195 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:02:29,434 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:02:30,207 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:02:30,228 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:02:31,422 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:02:32,273 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:02:32,352 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:02:32,806 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:02:33,517 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:02:33,710 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:02:34,630 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:02:35,232 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:02:35,453 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:02:35,512 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:02:35,515 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:02:35,517 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:02:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:02:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:05.848675+00:00 (in 29.999468 seconds) gcworker stdout | 2023-01-17 05:02:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:05 UTC)" (scheduled at 2023-01-17 05:02:35.848675+00:00) gcworker stdout | 2023-01-17 05:02:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722155849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:02:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:02:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:02:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:02:36,605 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:02:36,713 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:36,722 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:36,722 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:36,728 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:37,206 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:02:37,626 [241] [DEBUG] [app] Starting request: urn:request:c87be6b9-b17c-4f8f-9d9a-a560a5967356 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:37,626 [239] [DEBUG] [app] Starting request: urn:request:33abb91f-f10d-47bc-809f-347b3352386a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:37,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:37,631 [242] [DEBUG] [app] Starting request: urn:request:c0599281-55f6-4e59-8ea1-3e43b3e0bdd3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:37,631 [232] [DEBUG] [app] Starting request: urn:request:684b681d-6c86-4de7-8529-14eb82533df3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:37,632 [242] [DEBUG] [app] Ending request: urn:request:c0599281-55f6-4e59-8ea1-3e43b3e0bdd3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0599281-55f6-4e59-8ea1-3e43b3e0bdd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:37,632 [232] [DEBUG] [app] Ending request: urn:request:684b681d-6c86-4de7-8529-14eb82533df3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:684b681d-6c86-4de7-8529-14eb82533df3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:02:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:02:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:37,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:37,637 [244] [DEBUG] [app] Starting request: urn:request:ed9e69b1-c61c-4084-a03a-6461f77b2e1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:37,637 [243] [DEBUG] [app] Starting request: urn:request:471c15a0-ce08-42f6-bd70-61ad7017061a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:37,637 [244] [DEBUG] [app] Ending request: urn:request:ed9e69b1-c61c-4084-a03a-6461f77b2e1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed9e69b1-c61c-4084-a03a-6461f77b2e1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:37,637 [243] [DEBUG] [app] Ending request: urn:request:471c15a0-ce08-42f6-bd70-61ad7017061a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:471c15a0-ce08-42f6-bd70-61ad7017061a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:37,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:37,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:37,648 [241] [DEBUG] [app] Ending request: urn:request:c87be6b9-b17c-4f8f-9d9a-a560a5967356 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c87be6b9-b17c-4f8f-9d9a-a560a5967356', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:37,648 [239] [DEBUG] [app] Ending request: urn:request:33abb91f-f10d-47bc-809f-347b3352386a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33abb91f-f10d-47bc-809f-347b3352386a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:37,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:02:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:37,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:02:37,805 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:02:37,807 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:02:37,812 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:02:37,817 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:02:37,819 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:02:37,822 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:02:37,826 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:02:37,841 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:02:37,843 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:02:37,980 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:02:38,038 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:02:38,055 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:02:38,085 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:02:38,109 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:02:38,114 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:02:38,116 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:02:38,120 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:02:38,123 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:02:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:43.935629+00:00 (in 5.200098 seconds) notificationworker stdout | 2023-01-17 05:02:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:48 UTC)" (scheduled at 2023-01-17 05:02:38.735066+00:00) notificationworker stdout | 2023-01-17 05:02:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 38, 735977), True, datetime.datetime(2023, 1, 17, 5, 2, 38, 735977), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:02:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:02:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:48.738458+00:00 (in 9.707201 seconds) repositorygcworker stdout | 2023-01-17 05:02:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:39 UTC)" (scheduled at 2023-01-17 05:02:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:02:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:02:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 39, 31695), True, datetime.datetime(2023, 1, 17, 5, 2, 39, 31695), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:02:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:02:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:02:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:02:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:52.837365+00:00 (in 13.606875 seconds) securityworker stdout | 2023-01-17 05:02:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:09 UTC)" (scheduled at 2023-01-17 05:02:39.230038+00:00) securityworker stdout | 2023-01-17 05:02:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:02:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:02:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:02:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:02:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:02:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:02:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:48.735066+00:00 (in 4.798918 seconds) notificationworker stdout | 2023-01-17 05:02:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:43 UTC)" (scheduled at 2023-01-17 05:02:43.935629+00:00) notificationworker stdout | 2023-01-17 05:02:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:02:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:43 UTC)" executed successfully builder stdout | 2023-01-17 05:02:46,736 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:46,745 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:46,745 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:46,752 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:02:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:02:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:59.337413+00:00 (in 11.404468 seconds) namespacegcworker stdout | 2023-01-17 05:02:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:47 UTC)" (scheduled at 2023-01-17 05:02:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:02:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:02:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:58.735066+00:00 (in 9.999476 seconds) notificationworker stdout | 2023-01-17 05:02:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:58 UTC)" (scheduled at 2023-01-17 05:02:48.735066+00:00) notificationworker stdout | 2023-01-17 05:02:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 48, 736031), True, datetime.datetime(2023, 1, 17, 5, 2, 48, 736031), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:02:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:02:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:39.030814+00:00 (in 50.291898 seconds) repositorygcworker stdout | 2023-01-17 05:02:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:48 UTC)" (scheduled at 2023-01-17 05:02:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:02:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:02:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:02:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:02:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:24.641816+00:00 (in 35.811071 seconds) exportactionlogsworker stdout | 2023-01-17 05:02:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:48 UTC)" (scheduled at 2023-01-17 05:02:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:02:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:02:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 48, 831150), True, datetime.datetime(2023, 1, 17, 5, 2, 48, 831150), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:02:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:02:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:02:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:02:52,626 [239] [DEBUG] [app] Starting request: urn:request:e87cbad9-c721-4a42-8a4e-6bd3986d9056 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:52,626 [244] [DEBUG] [app] Starting request: urn:request:0f1af68c-082d-498c-91de-d11e42338dfa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:02:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:52,631 [232] [DEBUG] [app] Starting request: urn:request:d52318d2-0901-4d9b-992e-45e7a93c4837 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,631 [242] [DEBUG] [app] Starting request: urn:request:7ce03bd3-1d7c-411a-a477-35e16ca97c56 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,632 [232] [DEBUG] [app] Ending request: urn:request:d52318d2-0901-4d9b-992e-45e7a93c4837 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d52318d2-0901-4d9b-992e-45e7a93c4837', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,632 [242] [DEBUG] [app] Ending request: urn:request:7ce03bd3-1d7c-411a-a477-35e16ca97c56 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ce03bd3-1d7c-411a-a477-35e16ca97c56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:02:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:02:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:52,636 [243] [DEBUG] [app] Starting request: urn:request:4b0a6cb6-82bd-48cc-a128-a60d4143c1a1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:52,637 [241] [DEBUG] [app] Starting request: urn:request:abb736db-1bb9-4e9c-b36d-342bb4e595d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:52,637 [243] [DEBUG] [app] Ending request: urn:request:4b0a6cb6-82bd-48cc-a128-a60d4143c1a1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b0a6cb6-82bd-48cc-a128-a60d4143c1a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:02:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:52,637 [241] [DEBUG] [app] Ending request: urn:request:abb736db-1bb9-4e9c-b36d-342bb4e595d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abb736db-1bb9-4e9c-b36d-342bb4e595d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:52,648 [244] [DEBUG] [app] Ending request: urn:request:0f1af68c-082d-498c-91de-d11e42338dfa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f1af68c-082d-498c-91de-d11e42338dfa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:52,648 [239] [DEBUG] [app] Ending request: urn:request:e87cbad9-c721-4a42-8a4e-6bd3986d9056 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e87cbad9-c721-4a42-8a4e-6bd3986d9056', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:02:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:09.230038+00:00 (in 16.392220 seconds) securityworker stdout | 2023-01-17 05:02:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:22 UTC)" (scheduled at 2023-01-17 05:02:52.837365+00:00) securityworker stdout | 2023-01-17 05:02:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:52,844 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:22 UTC)" executed successfully builder stdout | 2023-01-17 05:02:56,762 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:56,770 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:56,771 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:56,777 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:02:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:02:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:27.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 05:02:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:27 UTC)" (scheduled at 2023-01-17 05:02:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:02:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 2, 57, 227198), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:02:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:02:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:02:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:08.735066+00:00 (in 9.999458 seconds) notificationworker stdout | 2023-01-17 05:02:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:08 UTC)" (scheduled at 2023-01-17 05:02:58.735066+00:00) notificationworker stdout | 2023-01-17 05:02:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 58, 736038), True, datetime.datetime(2023, 1, 17, 5, 2, 58, 736038), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:02:58,933 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:02:59,231 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} namespacegcworker stdout | 2023-01-17 05:02:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:02:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:47.932490+00:00 (in 48.594618 seconds) namespacegcworker stdout | 2023-01-17 05:02:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:59 UTC)" (scheduled at 2023-01-17 05:02:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:02:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:02:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 59, 338258), True, datetime.datetime(2023, 1, 17, 5, 2, 59, 338258), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:02:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:02:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:02:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:59 UTC)" executed successfully servicekey stdout | 2023-01-17 05:02:59,469 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:03:00,244 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:03:00,263 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:03:01,457 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:03:02,288 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:03:02,388 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:03:02,814 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:03:02,827 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 05:03:02,827 [67] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2023-01-17 05:03:02 UTC)" (scheduled at 2023-01-17 05:03:02.826956+00:00) globalpromstats stdout | 2023-01-17 05:03:02,827 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:02.826956+00:00 (in 3599.999631 seconds) globalpromstats stdout | 2023-01-17 05:03:02,827 [67] [DEBUG] [__main__] Attempting to report stats globalpromstats stdout | 2023-01-17 05:03:02,828 [67] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:03:02,828 [67] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ... globalpromstats stdout | 2023-01-17 05:03:02,828 [67] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2023-01-17 05:03:02,828 [67] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:03:02,828 [67] [DEBUG] [__main__] Reporting global stats globalpromstats stdout | 2023-01-17 05:03:02,829 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2023-01-17 05:03:02,836 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False]) globalpromstats stdout | 2023-01-17 05:03:02,838 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True]) globalpromstats stdout | 2023-01-17 05:03:02,840 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2023-01-17 05:03:02,841 [67] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 05:03:02,841 [67] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:03:02,841 [67] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2023-01-17 05:03:02,841 [67] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:03:02,841 [67] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2023-01-17 06:03:02 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:03:03,547 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:03:03,744 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:03:04,665 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:03:05,268 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:03:05,464 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:03:05,517 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:03:05,521 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:03:05,523 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:03:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:03:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:35.848675+00:00 (in 29.999469 seconds) gcworker stdout | 2023-01-17 05:03:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:35 UTC)" (scheduled at 2023-01-17 05:03:05.848675+00:00) gcworker stdout | 2023-01-17 05:03:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722185849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:03:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:03:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:03:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:03:06,642 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:03:06,781 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:06,789 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:06,789 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:06,795 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:07,217 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:03:07,626 [239] [DEBUG] [app] Starting request: urn:request:3b7d6e7d-ac54-4530-a493-bd80318eb53c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:07,626 [243] [DEBUG] [app] Starting request: urn:request:029cf864-b25b-4844-aeaf-303941124848 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:07,631 [242] [DEBUG] [app] Starting request: urn:request:fc07e236-2055-48c4-a182-53b1e71fe291 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,631 [236] [DEBUG] [app] Starting request: urn:request:b94e9050-542b-4d49-add3-dd6d21f46f53 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,631 [242] [DEBUG] [app] Ending request: urn:request:fc07e236-2055-48c4-a182-53b1e71fe291 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc07e236-2055-48c4-a182-53b1e71fe291', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,631 [236] [DEBUG] [app] Ending request: urn:request:b94e9050-542b-4d49-add3-dd6d21f46f53 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b94e9050-542b-4d49-add3-dd6d21f46f53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:03:07,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:03:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:07,636 [239] [DEBUG] [app] Starting request: urn:request:0e5fd7c7-0a27-46e9-838e-41aa8b853bcd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:07,637 [239] [DEBUG] [app] Ending request: urn:request:0e5fd7c7-0a27-46e9-838e-41aa8b853bcd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0e5fd7c7-0a27-46e9-838e-41aa8b853bcd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:07,637 [244] [DEBUG] [app] Starting request: urn:request:5860b092-ae61-4c52-812c-37b75dae146f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:03:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:07,637 [244] [DEBUG] [app] Ending request: urn:request:5860b092-ae61-4c52-812c-37b75dae146f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5860b092-ae61-4c52-812c-37b75dae146f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:07,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:07,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:07,648 [243] [DEBUG] [app] Ending request: urn:request:029cf864-b25b-4844-aeaf-303941124848 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:029cf864-b25b-4844-aeaf-303941124848', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:07,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:03:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:03:07,650 [239] [DEBUG] [app] Ending request: urn:request:3b7d6e7d-ac54-4530-a493-bd80318eb53c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3b7d6e7d-ac54-4530-a493-bd80318eb53c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:07,650 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) gunicorn-web stdout | 2023-01-17 05:03:07,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:03:07,812 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:03:07,814 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:03:07,819 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:03:07,823 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:03:07,826 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:03:07,829 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:03:07,833 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:03:07,847 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:03:07,851 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:03:07,987 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:03:08,052 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:03:08,089 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:03:08,112 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:03:08,115 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:03:08,119 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:03:08,122 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:03:08,126 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:03:08,128 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:03:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:18.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 05:03:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:18 UTC)" (scheduled at 2023-01-17 05:03:08.735066+00:00) notificationworker stdout | 2023-01-17 05:03:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 8, 735928), True, datetime.datetime(2023, 1, 17, 5, 3, 8, 735928), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:03:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:22.837365+00:00 (in 13.606844 seconds) securityworker stdout | 2023-01-17 05:03:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:39 UTC)" (scheduled at 2023-01-17 05:03:09.230038+00:00) securityworker stdout | 2023-01-17 05:03:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:03:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:03:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:03:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:03:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:03:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:03:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:39 UTC)" executed successfully builder stdout | 2023-01-17 05:03:16,805 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:16,814 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:16,814 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:16,820 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:28.735066+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:03:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:28 UTC)" (scheduled at 2023-01-17 05:03:18.735066+00:00) notificationworker stdout | 2023-01-17 05:03:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 18, 735951), True, datetime.datetime(2023, 1, 17, 5, 3, 18, 735951), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:03:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:03:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:25.334138+00:00 (in 5.091479 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:03:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:20 UTC)" (scheduled at 2023-01-17 05:03:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:03:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:03:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 20, 243089), True, datetime.datetime(2023, 1, 17, 5, 3, 20, 243089), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:03:20,253 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:03:20,253 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:03:20,253 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:03:22,625 [244] [DEBUG] [app] Starting request: urn:request:5834ff3f-0807-4d39-926a-b6afadc0286a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:22,625 [241] [DEBUG] [app] Starting request: urn:request:dac71322-1efa-4879-85fe-e19c129e2901 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:22,630 [242] [DEBUG] [app] Starting request: urn:request:bada7292-ac2e-451a-9e45-07f9673fe0db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:22,631 [232] [DEBUG] [app] Starting request: urn:request:abddeb2e-2b03-4bb8-a80d-b7677cc09faa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:22,631 [242] [DEBUG] [app] Ending request: urn:request:bada7292-ac2e-451a-9e45-07f9673fe0db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bada7292-ac2e-451a-9e45-07f9673fe0db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:22,631 [232] [DEBUG] [app] Ending request: urn:request:abddeb2e-2b03-4bb8-a80d-b7677cc09faa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abddeb2e-2b03-4bb8-a80d-b7677cc09faa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:03:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:03:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:22,635 [239] [DEBUG] [app] Starting request: urn:request:47eb0bd8-7a07-43d5-9372-4927efb9fc93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:22,636 [244] [DEBUG] [app] Starting request: urn:request:6694f9f3-d8d0-415f-b281-a13ca35a8b91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:22,636 [244] [DEBUG] [app] Ending request: urn:request:6694f9f3-d8d0-415f-b281-a13ca35a8b91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6694f9f3-d8d0-415f-b281-a13ca35a8b91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:22,636 [239] [DEBUG] [app] Ending request: urn:request:47eb0bd8-7a07-43d5-9372-4927efb9fc93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47eb0bd8-7a07-43d5-9372-4927efb9fc93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:03:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:22,647 [244] [DEBUG] [app] Ending request: urn:request:5834ff3f-0807-4d39-926a-b6afadc0286a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5834ff3f-0807-4d39-926a-b6afadc0286a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:22,647 [241] [DEBUG] [app] Ending request: urn:request:dac71322-1efa-4879-85fe-e19c129e2901 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dac71322-1efa-4879-85fe-e19c129e2901', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:22,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:03:22,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) securityworker stdout | 2023-01-17 05:03:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:39.230038+00:00 (in 16.392233 seconds) securityworker stdout | 2023-01-17 05:03:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:52 UTC)" (scheduled at 2023-01-17 05:03:22.837365+00:00) securityworker stdout | 2023-01-17 05:03:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:03:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:03:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:48.830274+00:00 (in 24.187992 seconds) exportactionlogsworker stdout | 2023-01-17 05:03:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:24 UTC)" (scheduled at 2023-01-17 05:03:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:03:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:03:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:03:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:03:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:20.242198+00:00 (in 54.907580 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:03:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:25 UTC)" (scheduled at 2023-01-17 05:03:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:03:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:03:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:25 UTC)" executed successfully builder stdout | 2023-01-17 05:03:26,823 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:26,832 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:26,832 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:26,838 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:03:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:03:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:57.226424+00:00 (in 29.999528 seconds) buildlogsarchiver stdout | 2023-01-17 05:03:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:57 UTC)" (scheduled at 2023-01-17 05:03:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:03:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 3, 27, 227285), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:03:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:03:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:03:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:38.735066+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:03:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:38 UTC)" (scheduled at 2023-01-17 05:03:28.735066+00:00) notificationworker stdout | 2023-01-17 05:03:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 28, 736050), True, datetime.datetime(2023, 1, 17, 5, 3, 28, 736050), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:03:28,968 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:03:29,266 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:03:29,482 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:03:30,273 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:03:30,276 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:03:31,468 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:03:32,317 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:03:32,423 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:03:32,832 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:03:33,567 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:03:33,780 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:03:34,701 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:03:35,281 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:03:35,483 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:03:35,523 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:03:35,527 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:03:35,530 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:03:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:03:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:05.848675+00:00 (in 29.999493 seconds) gcworker stdout | 2023-01-17 05:03:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:05 UTC)" (scheduled at 2023-01-17 05:03:35.848675+00:00) gcworker stdout | 2023-01-17 05:03:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722215849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:03:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:03:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:03:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:03:36,678 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:03:36,848 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:36,857 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:36,857 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:36,862 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:37,252 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:03:37,625 [239] [DEBUG] [app] Starting request: urn:request:26f74d21-7120-44cc-8476-dd6ab6c170d4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:37,625 [243] [DEBUG] [app] Starting request: urn:request:f3e30c82-3927-48e8-a248-399ee2bad0ab (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:37,630 [242] [DEBUG] [app] Starting request: urn:request:a712f6a5-5b1b-4b96-8d2f-1dfe8442ba51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,631 [232] [DEBUG] [app] Starting request: urn:request:18c7db72-82af-4738-9531-e82298aaaaab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,631 [242] [DEBUG] [app] Ending request: urn:request:a712f6a5-5b1b-4b96-8d2f-1dfe8442ba51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a712f6a5-5b1b-4b96-8d2f-1dfe8442ba51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:03:37,631 [232] [DEBUG] [app] Ending request: urn:request:18c7db72-82af-4738-9531-e82298aaaaab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18c7db72-82af-4738-9531-e82298aaaaab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:03:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:03:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:37,636 [244] [DEBUG] [app] Starting request: urn:request:07432572-7f44-4d10-aaa9-e38fd6b062e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:37,636 [239] [DEBUG] [app] Starting request: urn:request:eec37506-b21a-4e06-8c22-b7cc7524a205 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:37,636 [244] [DEBUG] [app] Ending request: urn:request:07432572-7f44-4d10-aaa9-e38fd6b062e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:07432572-7f44-4d10-aaa9-e38fd6b062e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:37,636 [239] [DEBUG] [app] Ending request: urn:request:eec37506-b21a-4e06-8c22-b7cc7524a205 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eec37506-b21a-4e06-8c22-b7cc7524a205', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:37,647 [239] [DEBUG] [app] Ending request: urn:request:26f74d21-7120-44cc-8476-dd6ab6c170d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26f74d21-7120-44cc-8476-dd6ab6c170d4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:37,647 [243] [DEBUG] [app] Ending request: urn:request:f3e30c82-3927-48e8-a248-399ee2bad0ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3e30c82-3927-48e8-a248-399ee2bad0ab', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:03:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:03:37,820 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:03:37,823 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:03:37,827 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:03:37,830 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:03:37,832 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:03:37,838 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:03:37,841 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:03:37,853 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:03:37,858 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:03:37,993 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:03:38,088 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 05:03:38,122 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} expiredappspecifictokenworker stdout | 2023-01-17 05:03:38,124 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 05:03:38,127 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:03:38,129 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:03:38,131 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:03:38,135 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} blobuploadcleanupworker stdout | 2023-01-17 05:03:38,146 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:03:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:43.935629+00:00 (in 5.200054 seconds) notificationworker stdout | 2023-01-17 05:03:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:48 UTC)" (scheduled at 2023-01-17 05:03:38.735066+00:00) notificationworker stdout | 2023-01-17 05:03:38,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 38, 736054), True, datetime.datetime(2023, 1, 17, 5, 3, 38, 736054), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:03:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:03:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:48.738458+00:00 (in 9.707186 seconds) repositorygcworker stdout | 2023-01-17 05:03:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:39 UTC)" (scheduled at 2023-01-17 05:03:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:03:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:03:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 39, 31742), True, datetime.datetime(2023, 1, 17, 5, 3, 39, 31742), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:03:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:03:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:03:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:03:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:52.837365+00:00 (in 13.606841 seconds) securityworker stdout | 2023-01-17 05:03:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:09 UTC)" (scheduled at 2023-01-17 05:03:39.230038+00:00) securityworker stdout | 2023-01-17 05:03:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:03:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:03:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:03:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:48.735066+00:00 (in 4.798964 seconds) notificationworker stdout | 2023-01-17 05:03:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:43 UTC)" (scheduled at 2023-01-17 05:03:43.935629+00:00) notificationworker stdout | 2023-01-17 05:03:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:03:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:43 UTC)" executed successfully builder stdout | 2023-01-17 05:03:46,873 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:46,882 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:46,882 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:46,888 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:03:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:03:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:59.337413+00:00 (in 11.404482 seconds) namespacegcworker stdout | 2023-01-17 05:03:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:47 UTC)" (scheduled at 2023-01-17 05:03:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:03:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:03:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:56.935055+00:00 (in 8.199513 seconds) notificationworker stdout | 2023-01-17 05:03:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:58 UTC)" (scheduled at 2023-01-17 05:03:48.735066+00:00) notificationworker stdout | 2023-01-17 05:03:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 48, 735982), True, datetime.datetime(2023, 1, 17, 5, 3, 48, 735982), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:03:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:03:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:39.030814+00:00 (in 50.291880 seconds) repositorygcworker stdout | 2023-01-17 05:03:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:48 UTC)" (scheduled at 2023-01-17 05:03:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:03:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:03:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:03:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:03:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:24.641816+00:00 (in 35.811068 seconds) exportactionlogsworker stdout | 2023-01-17 05:03:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:48 UTC)" (scheduled at 2023-01-17 05:03:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:03:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:03:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 48, 831167), True, datetime.datetime(2023, 1, 17, 5, 3, 48, 831167), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:03:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:03:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:03:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:03:52,625 [244] [DEBUG] [app] Starting request: urn:request:5968656d-1b41-4a94-bbef-50102df4d89d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:52,625 [241] [DEBUG] [app] Starting request: urn:request:c793bc64-0d52-4983-9fd3-520afc62a2bb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:03:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:52,630 [242] [DEBUG] [app] Starting request: urn:request:864c0acc-79b9-4643-a29a-708f71831cba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,630 [236] [DEBUG] [app] Starting request: urn:request:f9114cef-f224-4b7b-b134-779c4a7cfdd8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,630 [242] [DEBUG] [app] Ending request: urn:request:864c0acc-79b9-4643-a29a-708f71831cba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:864c0acc-79b9-4643-a29a-708f71831cba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,630 [236] [DEBUG] [app] Ending request: urn:request:f9114cef-f224-4b7b-b134-779c4a7cfdd8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9114cef-f224-4b7b-b134-779c4a7cfdd8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:03:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:03:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:52,635 [243] [DEBUG] [app] Starting request: urn:request:36a07aa7-00a5-4e32-b879-6615ff0ef38a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:52,636 [244] [DEBUG] [app] Starting request: urn:request:f9b9c2c2-d8f5-4bbe-8d83-d679771a6e2f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:52,636 [243] [DEBUG] [app] Ending request: urn:request:36a07aa7-00a5-4e32-b879-6615ff0ef38a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36a07aa7-00a5-4e32-b879-6615ff0ef38a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:52,636 [244] [DEBUG] [app] Ending request: urn:request:f9b9c2c2-d8f5-4bbe-8d83-d679771a6e2f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9b9c2c2-d8f5-4bbe-8d83-d679771a6e2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:03:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:52,648 [244] [DEBUG] [app] Ending request: urn:request:5968656d-1b41-4a94-bbef-50102df4d89d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5968656d-1b41-4a94-bbef-50102df4d89d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:52,648 [241] [DEBUG] [app] Ending request: urn:request:c793bc64-0d52-4983-9fd3-520afc62a2bb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c793bc64-0d52-4983-9fd3-520afc62a2bb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:03:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:03:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:09.230038+00:00 (in 16.392205 seconds) securityworker stdout | 2023-01-17 05:03:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:22 UTC)" (scheduled at 2023-01-17 05:03:52.837365+00:00) securityworker stdout | 2023-01-17 05:03:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:22 UTC)" executed successfully builder stdout | 2023-01-17 05:03:56,898 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:56,907 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:56,907 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:56,913 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:58.735066+00:00 (in 1.799518 seconds) notificationworker stdout | 2023-01-17 05:03:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:08:56 UTC)" (scheduled at 2023-01-17 05:03:56.935055+00:00) notificationworker stdout | 2023-01-17 05:03:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 3, 56, 935925), 'notification/%']) notificationworker stdout | 2023-01-17 05:03:56,942 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 3, 56, 935925), True, datetime.datetime(2023, 1, 17, 5, 3, 56, 935925), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:03:56,944 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 3, 56, 935925), True, datetime.datetime(2023, 1, 17, 5, 3, 56, 935925), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 3, 56, 935925), 'notification/%']) notificationworker stdout | 2023-01-17 05:03:56,946 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:56,946 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:08:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:03:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:03:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:27.226424+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:03:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:27 UTC)" (scheduled at 2023-01-17 05:03:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:03:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 3, 57, 227248), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:03:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:03:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:03:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:08.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:03:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:08 UTC)" (scheduled at 2023-01-17 05:03:58.735066+00:00) notificationworker stdout | 2023-01-17 05:03:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 58, 736006), True, datetime.datetime(2023, 1, 17, 5, 3, 58, 736006), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:03:58,997 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:03:59,284 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} namespacegcworker stdout | 2023-01-17 05:03:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:03:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:15.931994+00:00 (in 16.594104 seconds) namespacegcworker stdout | 2023-01-17 05:03:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:59 UTC)" (scheduled at 2023-01-17 05:03:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:03:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:03:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 59, 338303), True, datetime.datetime(2023, 1, 17, 5, 3, 59, 338303), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:03:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:03:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:03:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:59 UTC)" executed successfully servicekey stdout | 2023-01-17 05:03:59,518 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:04:00,309 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:04:00,312 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:04:01,504 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:04:02,332 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:04:02,441 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:04:02,867 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:04:03,585 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:04:03,786 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:04:04,730 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:04:05,317 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:04:05,518 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:04:05,528 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:04:05,533 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:04:05,535 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:04:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:04:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:35.848675+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 05:04:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:35 UTC)" (scheduled at 2023-01-17 05:04:05.848675+00:00) gcworker stdout | 2023-01-17 05:04:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722245849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:04:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:04:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:04:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:04:06,708 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:04:06,923 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:06,932 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:06,932 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:06,938 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:07,263 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:04:07,625 [243] [DEBUG] [app] Starting request: urn:request:c3dd6b7e-49f9-4821-9ad9-4d7c0fb79bd8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:07,625 [244] [DEBUG] [app] Starting request: urn:request:c0c4daa3-d1cf-4e0b-a445-fba760ecd8a5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:07,630 [242] [DEBUG] [app] Starting request: urn:request:aa7682d5-cd0f-430b-88cb-b08faa1c09e0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,630 [236] [DEBUG] [app] Starting request: urn:request:41552682-9edd-47ab-ae79-5712ad69fdf5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,630 [242] [DEBUG] [app] Ending request: urn:request:aa7682d5-cd0f-430b-88cb-b08faa1c09e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa7682d5-cd0f-430b-88cb-b08faa1c09e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:04:07,631 [236] [DEBUG] [app] Ending request: urn:request:41552682-9edd-47ab-ae79-5712ad69fdf5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41552682-9edd-47ab-ae79-5712ad69fdf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:04:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:04:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:07,635 [244] [DEBUG] [app] Starting request: urn:request:6203dc95-4b9f-4e8a-8836-cc18e55ccfe6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:07,636 [244] [DEBUG] [app] Ending request: urn:request:6203dc95-4b9f-4e8a-8836-cc18e55ccfe6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6203dc95-4b9f-4e8a-8836-cc18e55ccfe6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:07,636 [241] [DEBUG] [app] Starting request: urn:request:36f27a80-95fb-4c18-9288-449313d52e51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:04:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:07,636 [241] [DEBUG] [app] Ending request: urn:request:36f27a80-95fb-4c18-9288-449313d52e51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36f27a80-95fb-4c18-9288-449313d52e51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:07,647 [244] [DEBUG] [app] Ending request: urn:request:c0c4daa3-d1cf-4e0b-a445-fba760ecd8a5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0c4daa3-d1cf-4e0b-a445-fba760ecd8a5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:07,647 [243] [DEBUG] [app] Ending request: urn:request:c3dd6b7e-49f9-4821-9ad9-4d7c0fb79bd8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3dd6b7e-49f9-4821-9ad9-4d7c0fb79bd8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:04:07,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:04:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:04:07,828 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:04:07,830 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:04:07,834 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:04:07,836 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:04:07,839 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:04:07,845 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:04:07,847 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:04:07,859 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:04:07,864 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:04:08,026 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:04:08,123 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 05:04:08,127 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:04:08,133 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:04:08,136 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:04:08,138 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} expiredappspecifictokenworker stdout | 2023-01-17 05:04:08,140 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 05:04:08,143 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} blobuploadcleanupworker stdout | 2023-01-17 05:04:08,175 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:04:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:18.735066+00:00 (in 9.999476 seconds) notificationworker stdout | 2023-01-17 05:04:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:18 UTC)" (scheduled at 2023-01-17 05:04:08.735066+00:00) notificationworker stdout | 2023-01-17 05:04:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 8, 735992), True, datetime.datetime(2023, 1, 17, 5, 4, 8, 735992), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:04:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:22.837365+00:00 (in 13.606837 seconds) securityworker stdout | 2023-01-17 05:04:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:39 UTC)" (scheduled at 2023-01-17 05:04:09.230038+00:00) securityworker stdout | 2023-01-17 05:04:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:04:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:04:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:09,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:04:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:39 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:04:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:04:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:47.932490+00:00 (in 32.000038 seconds) namespacegcworker stdout | 2023-01-17 05:04:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:09:15 UTC)" (scheduled at 2023-01-17 05:04:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:04:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 4, 15, 932790), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:04:15,939 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 4, 15, 932790), True, datetime.datetime(2023, 1, 17, 5, 4, 15, 932790), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:04:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 4, 15, 932790), True, datetime.datetime(2023, 1, 17, 5, 4, 15, 932790), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 4, 15, 932790), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:04:15,943 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:04:15,943 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:09:15 UTC)" executed successfully builder stdout | 2023-01-17 05:04:16,948 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:16,956 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:16,956 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:16,962 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:28.735066+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:04:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:28 UTC)" (scheduled at 2023-01-17 05:04:18.735066+00:00) notificationworker stdout | 2023-01-17 05:04:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 18, 735991), True, datetime.datetime(2023, 1, 17, 5, 4, 18, 735991), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:04:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:04:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:25.334138+00:00 (in 5.091500 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:04:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:20 UTC)" (scheduled at 2023-01-17 05:04:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:04:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:04:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 20, 243024), True, datetime.datetime(2023, 1, 17, 5, 4, 20, 243024), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:04:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:04:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:04:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:04:22,625 [239] [DEBUG] [app] Starting request: urn:request:dcada31d-6ece-41a5-8eaf-0aee6f8a86dd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:22,625 [244] [DEBUG] [app] Starting request: urn:request:ae233f13-00ea-443c-9b86-eeb89595ee56 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:22,630 [236] [DEBUG] [app] Starting request: urn:request:8caea506-df7c-48a4-ba7e-57cf93040fac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,630 [232] [DEBUG] [app] Starting request: urn:request:5bc21c3b-9998-48fc-ae8e-460abc098699 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,631 [236] [DEBUG] [app] Ending request: urn:request:8caea506-df7c-48a4-ba7e-57cf93040fac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8caea506-df7c-48a4-ba7e-57cf93040fac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:04:22,631 [232] [DEBUG] [app] Ending request: urn:request:5bc21c3b-9998-48fc-ae8e-460abc098699 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5bc21c3b-9998-48fc-ae8e-460abc098699', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:04:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:04:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:22,635 [244] [DEBUG] [app] Starting request: urn:request:ffeb2241-c358-4307-b4c2-ce5116ade172 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:22,635 [244] [DEBUG] [app] Ending request: urn:request:ffeb2241-c358-4307-b4c2-ce5116ade172 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ffeb2241-c358-4307-b4c2-ce5116ade172', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:22,637 [243] [DEBUG] [app] Starting request: urn:request:9048319b-a310-4d6c-a3b1-aafd4f457ac9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:22,637 [243] [DEBUG] [app] Ending request: urn:request:9048319b-a310-4d6c-a3b1-aafd4f457ac9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9048319b-a310-4d6c-a3b1-aafd4f457ac9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:22,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:22,647 [239] [DEBUG] [app] Ending request: urn:request:dcada31d-6ece-41a5-8eaf-0aee6f8a86dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcada31d-6ece-41a5-8eaf-0aee6f8a86dd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:22,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:04:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:04:22,648 [244] [DEBUG] [app] Ending request: urn:request:ae233f13-00ea-443c-9b86-eeb89595ee56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ae233f13-00ea-443c-9b86-eeb89595ee56', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:04:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:39.230038+00:00 (in 16.392223 seconds) securityworker stdout | 2023-01-17 05:04:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:52 UTC)" (scheduled at 2023-01-17 05:04:22.837365+00:00) securityworker stdout | 2023-01-17 05:04:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:04:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:04:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:48.830274+00:00 (in 24.188014 seconds) exportactionlogsworker stdout | 2023-01-17 05:04:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:24 UTC)" (scheduled at 2023-01-17 05:04:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:04:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:04:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:04:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:04:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:20.242198+00:00 (in 54.907581 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:04:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:25 UTC)" (scheduled at 2023-01-17 05:04:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:04:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:04:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:25 UTC)" executed successfully builder stdout | 2023-01-17 05:04:26,972 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:26,981 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:26,981 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:26,987 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:04:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:04:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:57.226424+00:00 (in 29.999576 seconds) buildlogsarchiver stdout | 2023-01-17 05:04:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:57 UTC)" (scheduled at 2023-01-17 05:04:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:04:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 4, 27, 227195), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:04:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:04:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:04:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:38.735066+00:00 (in 9.999491 seconds) notificationworker stdout | 2023-01-17 05:04:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:38 UTC)" (scheduled at 2023-01-17 05:04:28.735066+00:00) notificationworker stdout | 2023-01-17 05:04:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 28, 735981), True, datetime.datetime(2023, 1, 17, 5, 4, 28, 735981), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:04:29,033 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:04:29,305 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:04:29,541 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:04:30,332 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:04:30,341 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:04:31,523 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:04:32,339 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:04:32,454 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:04:32,877 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:04:33,592 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:04:33,807 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:04:34,761 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:04:35,353 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:04:35,536 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} queuecleanupworker stdout | 2023-01-17 05:04:35,539 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-secscan stdout | 2023-01-17 05:04:35,541 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:04:35,544 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:04:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:04:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:05.848675+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 05:04:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:05 UTC)" (scheduled at 2023-01-17 05:04:35.848675+00:00) gcworker stdout | 2023-01-17 05:04:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722275849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:04:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:04:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:04:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:04:36,720 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:04:36,997 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:37,007 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:37,007 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:37,013 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:37,273 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:04:37,625 [243] [DEBUG] [app] Starting request: urn:request:2964399c-7838-423b-a5e9-c17f60587f55 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:37,625 [241] [DEBUG] [app] Starting request: urn:request:122a9099-2239-4bbc-a8a3-248615186c1b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:37,630 [237] [DEBUG] [app] Starting request: urn:request:28a39b99-3a91-4cb2-9c1f-01488d7b01af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,631 [237] [DEBUG] [app] Ending request: urn:request:28a39b99-3a91-4cb2-9c1f-01488d7b01af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28a39b99-3a91-4cb2-9c1f-01488d7b01af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,631 [242] [DEBUG] [app] Starting request: urn:request:ae464367-3308-4787-8806-d0b7d270d138 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:04:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:04:37,631 [242] [DEBUG] [app] Ending request: urn:request:ae464367-3308-4787-8806-d0b7d270d138 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ae464367-3308-4787-8806-d0b7d270d138', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:04:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:37,635 [239] [DEBUG] [app] Starting request: urn:request:18d7a48a-a873-4800-adac-789a2b549b4a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:37,635 [239] [DEBUG] [app] Ending request: urn:request:18d7a48a-a873-4800-adac-789a2b549b4a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18d7a48a-a873-4800-adac-789a2b549b4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:37,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:37,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:37,636 [239] [DEBUG] [app] Starting request: urn:request:2e715485-56a6-4bcb-9754-3c9ae86cd416 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:37,637 [239] [DEBUG] [app] Ending request: urn:request:2e715485-56a6-4bcb-9754-3c9ae86cd416 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e715485-56a6-4bcb-9754-3c9ae86cd416', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:37,646 [243] [DEBUG] [app] Ending request: urn:request:2964399c-7838-423b-a5e9-c17f60587f55 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2964399c-7838-423b-a5e9-c17f60587f55', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:37,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:37,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:04:37,648 [241] [DEBUG] [app] Ending request: urn:request:122a9099-2239-4bbc-a8a3-248615186c1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:122a9099-2239-4bbc-a8a3-248615186c1b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-registry stdout | 2023-01-17 05:04:37,836 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:04:37,839 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:04:37,841 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:04:37,843 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:04:37,846 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:04:37,851 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:04:37,854 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:04:37,866 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:04:37,870 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:04:38,034 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:04:38,133 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} securityscanningnotificationworker stdout | 2023-01-17 05:04:38,139 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} gunicorn-web stdout | 2023-01-17 05:04:38,141 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:04:38,144 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:04:38,146 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:04:38,148 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:04:38,164 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:04:38,186 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:04:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:43.935629+00:00 (in 5.200075 seconds) notificationworker stdout | 2023-01-17 05:04:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:48 UTC)" (scheduled at 2023-01-17 05:04:38.735066+00:00) notificationworker stdout | 2023-01-17 05:04:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 38, 735933), True, datetime.datetime(2023, 1, 17, 5, 4, 38, 735933), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:04:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:04:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:48.738458+00:00 (in 9.707221 seconds) repositorygcworker stdout | 2023-01-17 05:04:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:39 UTC)" (scheduled at 2023-01-17 05:04:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:04:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:04:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 39, 31665), True, datetime.datetime(2023, 1, 17, 5, 4, 39, 31665), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:04:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:04:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:04:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:04:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:52.837365+00:00 (in 13.606906 seconds) securityworker stdout | 2023-01-17 05:04:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:09 UTC)" (scheduled at 2023-01-17 05:04:39.230038+00:00) securityworker stdout | 2023-01-17 05:04:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:04:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:04:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:04:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:48.735066+00:00 (in 4.798927 seconds) notificationworker stdout | 2023-01-17 05:04:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:43 UTC)" (scheduled at 2023-01-17 05:04:43.935629+00:00) notificationworker stdout | 2023-01-17 05:04:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:04:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:43 UTC)" executed successfully builder stdout | 2023-01-17 05:04:47,024 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:47,033 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:47,033 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:47,038 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:04:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:04:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:59.337413+00:00 (in 11.404495 seconds) namespacegcworker stdout | 2023-01-17 05:04:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:47 UTC)" (scheduled at 2023-01-17 05:04:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:04:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:04:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:58.735066+00:00 (in 9.999481 seconds) notificationworker stdout | 2023-01-17 05:04:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:58 UTC)" (scheduled at 2023-01-17 05:04:48.735066+00:00) notificationworker stdout | 2023-01-17 05:04:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 48, 735983), True, datetime.datetime(2023, 1, 17, 5, 4, 48, 735983), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:04:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:04:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:39.030814+00:00 (in 50.291899 seconds) repositorygcworker stdout | 2023-01-17 05:04:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:48 UTC)" (scheduled at 2023-01-17 05:04:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:04:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:04:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:04:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:04:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:24.641816+00:00 (in 35.811070 seconds) exportactionlogsworker stdout | 2023-01-17 05:04:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:48 UTC)" (scheduled at 2023-01-17 05:04:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:04:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:04:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 48, 831194), True, datetime.datetime(2023, 1, 17, 5, 4, 48, 831194), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:04:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:04:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:04:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:04:52,625 [239] [DEBUG] [app] Starting request: urn:request:5e0f84ec-32f7-43f3-9dfd-d39adefdcc6e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:52,625 [241] [DEBUG] [app] Starting request: urn:request:98fc1587-8ba0-4cbe-9831-17d0e145e87e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:04:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:52,630 [242] [DEBUG] [app] Starting request: urn:request:8003bdd4-75fb-4ee4-a6f0-aec3430b44c7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,630 [235] [DEBUG] [app] Starting request: urn:request:fd2b06ab-69fa-46eb-a9a1-6a1e566ccbff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,631 [242] [DEBUG] [app] Ending request: urn:request:8003bdd4-75fb-4ee4-a6f0-aec3430b44c7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8003bdd4-75fb-4ee4-a6f0-aec3430b44c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,631 [235] [DEBUG] [app] Ending request: urn:request:fd2b06ab-69fa-46eb-a9a1-6a1e566ccbff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd2b06ab-69fa-46eb-a9a1-6a1e566ccbff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:04:52,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:04:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:52,635 [244] [DEBUG] [app] Starting request: urn:request:be5f8b48-8597-471f-a2dd-73ac5fbcccc6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:52,636 [244] [DEBUG] [app] Ending request: urn:request:be5f8b48-8597-471f-a2dd-73ac5fbcccc6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be5f8b48-8597-471f-a2dd-73ac5fbcccc6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:52,636 [243] [DEBUG] [app] Starting request: urn:request:d1d65670-0aa3-4ecf-a873-17589d61412e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:52,637 [243] [DEBUG] [app] Ending request: urn:request:d1d65670-0aa3-4ecf-a873-17589d61412e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d1d65670-0aa3-4ecf-a873-17589d61412e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:52,647 [239] [DEBUG] [app] Ending request: urn:request:5e0f84ec-32f7-43f3-9dfd-d39adefdcc6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5e0f84ec-32f7-43f3-9dfd-d39adefdcc6e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:04:52,648 [241] [DEBUG] [app] Ending request: urn:request:98fc1587-8ba0-4cbe-9831-17d0e145e87e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:98fc1587-8ba0-4cbe-9831-17d0e145e87e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:04:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:04:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:09.230038+00:00 (in 16.392191 seconds) securityworker stdout | 2023-01-17 05:04:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:22 UTC)" (scheduled at 2023-01-17 05:04:52.837365+00:00) securityworker stdout | 2023-01-17 05:04:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:22 UTC)" executed successfully builder stdout | 2023-01-17 05:04:57,049 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:57,057 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:57,057 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:57,063 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:04:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:04:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:27.226424+00:00 (in 29.999566 seconds) buildlogsarchiver stdout | 2023-01-17 05:04:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:27 UTC)" (scheduled at 2023-01-17 05:04:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:04:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 4, 57, 227235), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:04:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:04:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:04:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:08.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:04:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:08 UTC)" (scheduled at 2023-01-17 05:04:58.735066+00:00) notificationworker stdout | 2023-01-17 05:04:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 58, 735972), True, datetime.datetime(2023, 1, 17, 5, 4, 58, 735972), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:04:59,046 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} namespacegcworker stdout | 2023-01-17 05:04:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:04:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:47.932490+00:00 (in 48.594644 seconds) namespacegcworker stdout | 2023-01-17 05:04:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:59 UTC)" (scheduled at 2023-01-17 05:04:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:04:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:04:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 59, 338208), True, datetime.datetime(2023, 1, 17, 5, 4, 59, 338208), 0, 'namespacegc/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:04:59,341 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} namespacegcworker stdout | 2023-01-17 05:04:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:04:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:04:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:59 UTC)" executed successfully servicekey stdout | 2023-01-17 05:04:59,577 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:05:00,348 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:05:00,370 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:05:01,558 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:05:02,374 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:05:02,489 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:05:02,912 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:05:03,628 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:05:03,844 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:05:04,773 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:05:05,388 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:05:05,542 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:05:05,546 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:05:05,550 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:05:05,552 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:05:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:05:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:35.848675+00:00 (in 29.999495 seconds) gcworker stdout | 2023-01-17 05:05:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:35 UTC)" (scheduled at 2023-01-17 05:05:05.848675+00:00) gcworker stdout | 2023-01-17 05:05:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722305849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:05:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:05:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:05:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:05:06,744 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:05:07,073 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:07,081 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:07,081 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:07,087 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:07,299 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:05:07,625 [244] [DEBUG] [app] Starting request: urn:request:4fc4af4d-bdee-47eb-b073-d3875a44d43f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:07,625 [241] [DEBUG] [app] Starting request: urn:request:faf486e0-fa74-4d45-9b3f-8b05d1e6b1da (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:07,630 [242] [DEBUG] [app] Starting request: urn:request:2737ffe7-afe3-46b7-b4ae-3b7cfa618f43 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,631 [240] [DEBUG] [app] Starting request: urn:request:0fe32c30-3b41-4b18-90cb-57483a07842e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,631 [242] [DEBUG] [app] Ending request: urn:request:2737ffe7-afe3-46b7-b4ae-3b7cfa618f43 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2737ffe7-afe3-46b7-b4ae-3b7cfa618f43', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:05:07,631 [240] [DEBUG] [app] Ending request: urn:request:0fe32c30-3b41-4b18-90cb-57483a07842e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0fe32c30-3b41-4b18-90cb-57483a07842e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:05:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:05:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:07,635 [243] [DEBUG] [app] Starting request: urn:request:a6a74dad-b731-407d-9456-38a8005b2145 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:07,636 [239] [DEBUG] [app] Starting request: urn:request:97cd29b5-7e87-4dbc-9f6b-d10f62cd4451 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:07,636 [243] [DEBUG] [app] Ending request: urn:request:a6a74dad-b731-407d-9456-38a8005b2145 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6a74dad-b731-407d-9456-38a8005b2145', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:07,636 [239] [DEBUG] [app] Ending request: urn:request:97cd29b5-7e87-4dbc-9f6b-d10f62cd4451 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97cd29b5-7e87-4dbc-9f6b-d10f62cd4451', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:05:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:07,647 [241] [DEBUG] [app] Ending request: urn:request:faf486e0-fa74-4d45-9b3f-8b05d1e6b1da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:faf486e0-fa74-4d45-9b3f-8b05d1e6b1da', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:07,647 [244] [DEBUG] [app] Ending request: urn:request:4fc4af4d-bdee-47eb-b073-d3875a44d43f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4fc4af4d-bdee-47eb-b073-d3875a44d43f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:07,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:05:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:05:07,843 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:05:07,846 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:05:07,849 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:05:07,852 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:05:07,854 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:05:07,857 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:05:07,860 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:05:07,873 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:05:07,876 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:05:08,069 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:05:08,140 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:05:08,147 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:05:08,150 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:05:08,152 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:05:08,154 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:05:08,173 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:05:08,176 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:05:08,196 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:05:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:18.735066+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:05:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:18 UTC)" (scheduled at 2023-01-17 05:05:08.735066+00:00) notificationworker stdout | 2023-01-17 05:05:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 8, 735980), True, datetime.datetime(2023, 1, 17, 5, 5, 8, 735980), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:05:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:22.837365+00:00 (in 13.606827 seconds) securityworker stdout | 2023-01-17 05:05:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:39 UTC)" (scheduled at 2023-01-17 05:05:09.230038+00:00) securityworker stdout | 2023-01-17 05:05:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:05:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:05:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:05:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:05:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:05:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:05:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:39 UTC)" executed successfully builder stdout | 2023-01-17 05:05:17,092 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:17,101 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:17,101 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:17,107 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:28.735066+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:05:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:28 UTC)" (scheduled at 2023-01-17 05:05:18.735066+00:00) notificationworker stdout | 2023-01-17 05:05:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 18, 736059), True, datetime.datetime(2023, 1, 17, 5, 5, 18, 736059), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:05:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:05:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:25.334138+00:00 (in 5.091443 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:05:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:20 UTC)" (scheduled at 2023-01-17 05:05:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:05:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:05:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 20, 243127), True, datetime.datetime(2023, 1, 17, 5, 5, 20, 243127), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:05:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:05:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:05:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:05:22,625 [241] [DEBUG] [app] Starting request: urn:request:15d9099c-0e7e-45ee-bf77-b231498ed37c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:22,625 [244] [DEBUG] [app] Starting request: urn:request:3988b9b4-f1ad-462b-9aa3-603a69a9c31b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:22,631 [240] [DEBUG] [app] Starting request: urn:request:f7645c64-55da-4cc3-a1e8-6c012ddb6a14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,631 [242] [DEBUG] [app] Starting request: urn:request:379dd4e7-3058-4bc6-b0a6-808e959df7e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,631 [240] [DEBUG] [app] Ending request: urn:request:f7645c64-55da-4cc3-a1e8-6c012ddb6a14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f7645c64-55da-4cc3-a1e8-6c012ddb6a14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,631 [242] [DEBUG] [app] Ending request: urn:request:379dd4e7-3058-4bc6-b0a6-808e959df7e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:379dd4e7-3058-4bc6-b0a6-808e959df7e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:05:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:05:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:22,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:22,636 [243] [DEBUG] [app] Starting request: urn:request:556bdc84-d29f-424e-af5e-afe7000fa105 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:22,637 [243] [DEBUG] [app] Ending request: urn:request:556bdc84-d29f-424e-af5e-afe7000fa105 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:556bdc84-d29f-424e-af5e-afe7000fa105', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:22,637 [239] [DEBUG] [app] Starting request: urn:request:3e89a66b-03a7-4c36-8f51-274c9db614e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:22,637 [239] [DEBUG] [app] Ending request: urn:request:3e89a66b-03a7-4c36-8f51-274c9db614e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e89a66b-03a7-4c36-8f51-274c9db614e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:05:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:22,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:22,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:22,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:22,648 [241] [DEBUG] [app] Ending request: urn:request:15d9099c-0e7e-45ee-bf77-b231498ed37c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15d9099c-0e7e-45ee-bf77-b231498ed37c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:22,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:22,648 [244] [DEBUG] [app] Ending request: urn:request:3988b9b4-f1ad-462b-9aa3-603a69a9c31b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3988b9b4-f1ad-462b-9aa3-603a69a9c31b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:22,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 05:05:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:05:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:39.230038+00:00 (in 16.392139 seconds) securityworker stdout | 2023-01-17 05:05:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:52 UTC)" (scheduled at 2023-01-17 05:05:22.837365+00:00) securityworker stdout | 2023-01-17 05:05:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:05:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:05:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:48.830274+00:00 (in 24.187987 seconds) exportactionlogsworker stdout | 2023-01-17 05:05:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:24 UTC)" (scheduled at 2023-01-17 05:05:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:05:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:05:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:05:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:05:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:20.242198+00:00 (in 54.907614 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:05:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:25 UTC)" (scheduled at 2023-01-17 05:05:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:05:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:05:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:25 UTC)" executed successfully builder stdout | 2023-01-17 05:05:27,117 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:27,126 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:27,126 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:27,132 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:05:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:05:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:57.226424+00:00 (in 29.999560 seconds) buildlogsarchiver stdout | 2023-01-17 05:05:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:57 UTC)" (scheduled at 2023-01-17 05:05:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:05:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 5, 27, 227197), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:05:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:05:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:05:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:38.735066+00:00 (in 9.999430 seconds) notificationworker stdout | 2023-01-17 05:05:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:38 UTC)" (scheduled at 2023-01-17 05:05:28.735066+00:00) notificationworker stdout | 2023-01-17 05:05:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 28, 736075), True, datetime.datetime(2023, 1, 17, 5, 5, 28, 736075), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:05:29,069 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:05:29,349 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:05:29,596 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:05:30,378 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:05:30,380 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:05:31,594 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:05:32,410 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:05:32,496 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:05:32,934 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:05:33,651 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:05:33,860 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:05:34,798 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:05:35,423 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:05:35,548 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:05:35,552 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:05:35,555 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:05:35,563 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:05:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:05:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:05.848675+00:00 (in 29.999461 seconds) gcworker stdout | 2023-01-17 05:05:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:05 UTC)" (scheduled at 2023-01-17 05:05:35.848675+00:00) gcworker stdout | 2023-01-17 05:05:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722335849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:05:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:05:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:05:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:05:36,778 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:05:37,132 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:37,141 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:37,141 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:37,147 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:37,319 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:05:37,625 [244] [DEBUG] [app] Starting request: urn:request:6fb78379-d720-45af-8831-7ce1a18aba5e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:37,626 [239] [DEBUG] [app] Starting request: urn:request:a26e15c2-eb72-46dc-af69-bb1c29b6a672 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:37,631 [240] [DEBUG] [app] Starting request: urn:request:3c719d6a-2a07-48a9-b79e-402c8058c340 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:37,631 [242] [DEBUG] [app] Starting request: urn:request:7dcf0115-d966-4cf6-af11-48a5a78e27f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:37,631 [240] [DEBUG] [app] Ending request: urn:request:3c719d6a-2a07-48a9-b79e-402c8058c340 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c719d6a-2a07-48a9-b79e-402c8058c340', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:05:37,632 [242] [DEBUG] [app] Ending request: urn:request:7dcf0115-d966-4cf6-af11-48a5a78e27f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7dcf0115-d966-4cf6-af11-48a5a78e27f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:05:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:05:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:37,636 [243] [DEBUG] [app] Starting request: urn:request:79ace0ca-9fc1-4cb6-b539-d9b777cba539 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:37,636 [243] [DEBUG] [app] Ending request: urn:request:79ace0ca-9fc1-4cb6-b539-d9b777cba539 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79ace0ca-9fc1-4cb6-b539-d9b777cba539', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:37,637 [241] [DEBUG] [app] Starting request: urn:request:df36e559-537a-40c7-8a87-972e5bd3a3f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:37,638 [241] [DEBUG] [app] Ending request: urn:request:df36e559-537a-40c7-8a87-972e5bd3a3f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df36e559-537a-40c7-8a87-972e5bd3a3f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:05:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:37,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:37,649 [239] [DEBUG] [app] Ending request: urn:request:a26e15c2-eb72-46dc-af69-bb1c29b6a672 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a26e15c2-eb72-46dc-af69-bb1c29b6a672', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:37,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 05:05:37,654 [244] [DEBUG] [app] Ending request: urn:request:6fb78379-d720-45af-8831-7ce1a18aba5e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6fb78379-d720-45af-8831-7ce1a18aba5e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:37,654 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 05:05:37,655 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:05:37,852 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:05:37,854 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:05:37,856 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:05:37,859 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:05:37,861 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:05:37,863 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:05:37,866 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:05:37,879 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:05:37,882 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:05:38,078 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:05:38,146 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:05:38,153 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:05:38,156 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:05:38,159 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:05:38,162 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityscanningnotificationworker stdout | 2023-01-17 05:05:38,200 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:05:38,202 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:05:38,224 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:05:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:43.935629+00:00 (in 5.200141 seconds) notificationworker stdout | 2023-01-17 05:05:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:48 UTC)" (scheduled at 2023-01-17 05:05:38.735066+00:00) notificationworker stdout | 2023-01-17 05:05:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 38, 735869), True, datetime.datetime(2023, 1, 17, 5, 5, 38, 735869), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:05:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:05:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:48.738458+00:00 (in 9.707246 seconds) repositorygcworker stdout | 2023-01-17 05:05:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:39 UTC)" (scheduled at 2023-01-17 05:05:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:05:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:05:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 39, 31622), True, datetime.datetime(2023, 1, 17, 5, 5, 39, 31622), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:05:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:05:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:05:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:05:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:52.837365+00:00 (in 13.606772 seconds) securityworker stdout | 2023-01-17 05:05:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:09 UTC)" (scheduled at 2023-01-17 05:05:39.230038+00:00) securityworker stdout | 2023-01-17 05:05:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:05:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:05:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:05:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:48.735066+00:00 (in 4.798963 seconds) notificationworker stdout | 2023-01-17 05:05:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:43 UTC)" (scheduled at 2023-01-17 05:05:43.935629+00:00) notificationworker stdout | 2023-01-17 05:05:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:05:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:43 UTC)" executed successfully builder stdout | 2023-01-17 05:05:47,157 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:47,166 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:47,166 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:47,172 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:05:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:05:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:59.337413+00:00 (in 11.404454 seconds) namespacegcworker stdout | 2023-01-17 05:05:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:47 UTC)" (scheduled at 2023-01-17 05:05:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:05:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:05:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:58.735066+00:00 (in 9.999469 seconds) notificationworker stdout | 2023-01-17 05:05:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:58 UTC)" (scheduled at 2023-01-17 05:05:48.735066+00:00) notificationworker stdout | 2023-01-17 05:05:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 48, 736021), True, datetime.datetime(2023, 1, 17, 5, 5, 48, 736021), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:05:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:05:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:39.030814+00:00 (in 50.291929 seconds) repositorygcworker stdout | 2023-01-17 05:05:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:48 UTC)" (scheduled at 2023-01-17 05:05:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:05:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:05:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:05:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:05:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:24.641816+00:00 (in 35.811124 seconds) exportactionlogsworker stdout | 2023-01-17 05:05:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:48 UTC)" (scheduled at 2023-01-17 05:05:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:05:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:05:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 48, 831046), True, datetime.datetime(2023, 1, 17, 5, 5, 48, 831046), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:05:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:05:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:05:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:05:52,625 [243] [DEBUG] [app] Starting request: urn:request:8533408a-95bc-4b4e-83e3-8ae9924ceb41 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:52,625 [244] [DEBUG] [app] Starting request: urn:request:9c5e44d6-89fb-4c49-92a0-13bd608f135e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:05:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:52,630 [240] [DEBUG] [app] Starting request: urn:request:e58b9bd3-81dc-4c59-94f2-1868268d232c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:52,630 [240] [DEBUG] [app] Ending request: urn:request:e58b9bd3-81dc-4c59-94f2-1868268d232c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e58b9bd3-81dc-4c59-94f2-1868268d232c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:52,630 [236] [DEBUG] [app] Starting request: urn:request:2911b9ab-adaa-4284-83b8-57c4cfb8aed0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:05:52,631 [236] [DEBUG] [app] Ending request: urn:request:2911b9ab-adaa-4284-83b8-57c4cfb8aed0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2911b9ab-adaa-4284-83b8-57c4cfb8aed0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:05:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:52,635 [241] [DEBUG] [app] Starting request: urn:request:ba8d331c-d6c7-4fdd-8df9-419e7a10bae5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:52,635 [241] [DEBUG] [app] Ending request: urn:request:ba8d331c-d6c7-4fdd-8df9-419e7a10bae5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba8d331c-d6c7-4fdd-8df9-419e7a10bae5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:52,635 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:52,637 [239] [DEBUG] [app] Starting request: urn:request:e2b5a0a8-8a0a-487a-b086-3407cb174fe8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:52,637 [239] [DEBUG] [app] Ending request: urn:request:e2b5a0a8-8a0a-487a-b086-3407cb174fe8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e2b5a0a8-8a0a-487a-b086-3407cb174fe8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:52,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:52,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:52,646 [244] [DEBUG] [app] Ending request: urn:request:9c5e44d6-89fb-4c49-92a0-13bd608f135e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c5e44d6-89fb-4c49-92a0-13bd608f135e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:52,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:05:52,648 [243] [DEBUG] [app] Ending request: urn:request:8533408a-95bc-4b4e-83e3-8ae9924ceb41 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8533408a-95bc-4b4e-83e3-8ae9924ceb41', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) securityworker stdout | 2023-01-17 05:05:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:09.230038+00:00 (in 16.392208 seconds) securityworker stdout | 2023-01-17 05:05:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:22 UTC)" (scheduled at 2023-01-17 05:05:52.837365+00:00) securityworker stdout | 2023-01-17 05:05:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:22 UTC)" executed successfully builder stdout | 2023-01-17 05:05:57,182 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:57,191 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:57,191 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:57,197 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:05:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:05:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:27.226424+00:00 (in 29.999579 seconds) buildlogsarchiver stdout | 2023-01-17 05:05:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:27 UTC)" (scheduled at 2023-01-17 05:05:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:05:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 5, 57, 227216), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:05:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:05:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:05:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:08.735066+00:00 (in 9.999494 seconds) notificationworker stdout | 2023-01-17 05:05:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:08 UTC)" (scheduled at 2023-01-17 05:05:58.735066+00:00) notificationworker stdout | 2023-01-17 05:05:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 58, 736012), True, datetime.datetime(2023, 1, 17, 5, 5, 58, 736012), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:05:59,105 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} namespacegcworker stdout | 2023-01-17 05:05:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:05:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:47.932490+00:00 (in 48.594598 seconds) namespacegcworker stdout | 2023-01-17 05:05:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:59 UTC)" (scheduled at 2023-01-17 05:05:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:05:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:05:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 59, 338334), True, datetime.datetime(2023, 1, 17, 5, 5, 59, 338334), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:05:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:05:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:05:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:59 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:05:59,356 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:05:59,631 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:06:00,408 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:06:00,415 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:06:01,622 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:06:02,446 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:06:02,531 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:06:02,944 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:06:03,686 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:06:03,896 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:06:04,833 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:06:05,441 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:06:05,554 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:06:05,558 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:06:05,561 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:06:05,580 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:06:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:06:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:35.848675+00:00 (in 29.999341 seconds) gcworker stdout | 2023-01-17 05:06:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:35 UTC)" (scheduled at 2023-01-17 05:06:05.848675+00:00) gcworker stdout | 2023-01-17 05:06:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722365849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:06:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:06:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:06:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:06:06,800 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:06:07,207 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:07,216 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:07,216 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:07,221 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:07,347 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:06:07,625 [244] [DEBUG] [app] Starting request: urn:request:2f7cc19b-0976-4ddc-8f31-f8fd6de6a01c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:07,625 [239] [DEBUG] [app] Starting request: urn:request:abf70f15-e06b-4cf7-8fad-b2452272eee3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:07,630 [232] [DEBUG] [app] Starting request: urn:request:57b068c7-a1ee-40d3-8600-fcd4968aa06f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:07,630 [232] [DEBUG] [app] Ending request: urn:request:57b068c7-a1ee-40d3-8600-fcd4968aa06f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57b068c7-a1ee-40d3-8600-fcd4968aa06f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:07,630 [242] [DEBUG] [app] Starting request: urn:request:5811ffe9-66cc-455a-9e53-bbe9cff53157 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:06:07,631 [242] [DEBUG] [app] Ending request: urn:request:5811ffe9-66cc-455a-9e53-bbe9cff53157 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5811ffe9-66cc-455a-9e53-bbe9cff53157', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:06:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:07,635 [244] [DEBUG] [app] Starting request: urn:request:45487235-a499-43bd-8a9b-91f8a5c8389b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:07,635 [243] [DEBUG] [app] Starting request: urn:request:3072bba1-13d6-41f4-a66d-3928c94e0067 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:07,636 [244] [DEBUG] [app] Ending request: urn:request:45487235-a499-43bd-8a9b-91f8a5c8389b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45487235-a499-43bd-8a9b-91f8a5c8389b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:07,636 [243] [DEBUG] [app] Ending request: urn:request:3072bba1-13d6-41f4-a66d-3928c94e0067 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3072bba1-13d6-41f4-a66d-3928c94e0067', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:06:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:07,646 [244] [DEBUG] [app] Ending request: urn:request:2f7cc19b-0976-4ddc-8f31-f8fd6de6a01c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f7cc19b-0976-4ddc-8f31-f8fd6de6a01c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:07,646 [239] [DEBUG] [app] Ending request: urn:request:abf70f15-e06b-4cf7-8fad-b2452272eee3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:abf70f15-e06b-4cf7-8fad-b2452272eee3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:06:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:06:07,860 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:06:07,863 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:06:07,866 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:06:07,868 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:06:07,871 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:06:07,873 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:06:07,876 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:06:07,885 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:06:07,891 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:06:08,085 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:06:08,152 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:06:08,159 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:06:08,162 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:06:08,165 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:06:08,167 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityscanningnotificationworker stdout | 2023-01-17 05:06:08,205 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:06:08,237 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:06:08,239 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:06:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:18.735066+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:06:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:18 UTC)" (scheduled at 2023-01-17 05:06:08.735066+00:00) notificationworker stdout | 2023-01-17 05:06:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 8, 735953), True, datetime.datetime(2023, 1, 17, 5, 6, 8, 735953), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:06:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:22.837365+00:00 (in 13.606831 seconds) securityworker stdout | 2023-01-17 05:06:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:39 UTC)" (scheduled at 2023-01-17 05:06:09.230038+00:00) securityworker stdout | 2023-01-17 05:06:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:06:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:06:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:09,247 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:09,248 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:06:09,248 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:39 UTC)" executed successfully builder stdout | 2023-01-17 05:06:17,226 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:17,235 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:17,235 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:17,241 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:28.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 05:06:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:28 UTC)" (scheduled at 2023-01-17 05:06:18.735066+00:00) notificationworker stdout | 2023-01-17 05:06:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 18, 735892), True, datetime.datetime(2023, 1, 17, 5, 6, 18, 735892), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:06:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:06:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:25.334138+00:00 (in 5.091451 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:06:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:20 UTC)" (scheduled at 2023-01-17 05:06:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:06:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:06:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 20, 243088), True, datetime.datetime(2023, 1, 17, 5, 6, 20, 243088), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:06:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:06:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:06:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:06:22,625 [243] [DEBUG] [app] Starting request: urn:request:3e007047-5857-44d3-8678-4016fbe7f740 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:22,625 [239] [DEBUG] [app] Starting request: urn:request:f8cf72f3-d992-40d6-b7e9-7176a4ea601c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:22,630 [232] [DEBUG] [app] Starting request: urn:request:232de7b7-a5cc-408a-9df6-67e53ddf8362 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,630 [242] [DEBUG] [app] Starting request: urn:request:be147f15-bd58-4c93-85f4-7c8fb178d694 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,631 [232] [DEBUG] [app] Ending request: urn:request:232de7b7-a5cc-408a-9df6-67e53ddf8362 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:232de7b7-a5cc-408a-9df6-67e53ddf8362', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,631 [242] [DEBUG] [app] Ending request: urn:request:be147f15-bd58-4c93-85f4-7c8fb178d694 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be147f15-bd58-4c93-85f4-7c8fb178d694', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:06:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:06:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:22,636 [244] [DEBUG] [app] Starting request: urn:request:e2ae66ae-fe86-442e-b3f8-c02af6c0eb3c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:22,636 [239] [DEBUG] [app] Starting request: urn:request:38b6e9bf-53aa-48fa-8655-412824c726da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:22,636 [244] [DEBUG] [app] Ending request: urn:request:e2ae66ae-fe86-442e-b3f8-c02af6c0eb3c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e2ae66ae-fe86-442e-b3f8-c02af6c0eb3c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:22,636 [239] [DEBUG] [app] Ending request: urn:request:38b6e9bf-53aa-48fa-8655-412824c726da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:38b6e9bf-53aa-48fa-8655-412824c726da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:06:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:06:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:22,648 [239] [DEBUG] [app] Ending request: urn:request:f8cf72f3-d992-40d6-b7e9-7176a4ea601c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f8cf72f3-d992-40d6-b7e9-7176a4ea601c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:22,648 [243] [DEBUG] [app] Ending request: urn:request:3e007047-5857-44d3-8678-4016fbe7f740 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3e007047-5857-44d3-8678-4016fbe7f740', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:06:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:06:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:39.230038+00:00 (in 16.392183 seconds) securityworker stdout | 2023-01-17 05:06:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:52 UTC)" (scheduled at 2023-01-17 05:06:22.837365+00:00) securityworker stdout | 2023-01-17 05:06:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:22,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:22,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:06:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:06:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:44.641497+00:00 (in 19.999234 seconds) exportactionlogsworker stdout | 2023-01-17 05:06:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:24 UTC)" (scheduled at 2023-01-17 05:06:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:06:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:06:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:06:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:06:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:39.333860+00:00 (in 13.999262 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:06:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:25 UTC)" (scheduled at 2023-01-17 05:06:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:06:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:06:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:06:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:06:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:57.226424+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 05:06:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:57 UTC)" (scheduled at 2023-01-17 05:06:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:06:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 6, 27, 227250), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:06:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:06:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:06:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:57 UTC)" executed successfully builder stdout | 2023-01-17 05:06:27,251 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:27,258 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:27,258 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:27,264 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:38.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:06:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:38 UTC)" (scheduled at 2023-01-17 05:06:28.735066+00:00) notificationworker stdout | 2023-01-17 05:06:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 28, 735919), True, datetime.datetime(2023, 1, 17, 5, 6, 28, 735919), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:06:29,141 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:06:29,391 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:06:29,666 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:06:30,425 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:06:30,440 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:06:31,658 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:06:32,481 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:06:32,556 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:06:32,980 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:06:33,713 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:06:33,931 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:06:34,848 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:06:35,471 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:06:35,560 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:06:35,564 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:06:35,566 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:06:35,614 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:06:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:06:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:05.848675+00:00 (in 29.999511 seconds) gcworker stdout | 2023-01-17 05:06:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:05 UTC)" (scheduled at 2023-01-17 05:06:35.848675+00:00) gcworker stdout | 2023-01-17 05:06:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722395849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:06:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:06:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:06:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:06:36,836 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:06:37,274 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:37,282 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:37,282 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:37,288 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:37,382 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:06:37,626 [243] [DEBUG] [app] Starting request: urn:request:eb0e8408-4e6d-4533-91e8-d501a705fc3f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:37,626 [244] [DEBUG] [app] Starting request: urn:request:1e70a056-576e-4625-a4a0-f2db4d051a11 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:37,631 [232] [DEBUG] [app] Starting request: urn:request:8dc57514-d854-4125-8156-1fa5c34b97ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,631 [242] [DEBUG] [app] Starting request: urn:request:381d52f5-dd27-4c45-9f32-2b86892aebd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,631 [232] [DEBUG] [app] Ending request: urn:request:8dc57514-d854-4125-8156-1fa5c34b97ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8dc57514-d854-4125-8156-1fa5c34b97ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:06:37,631 [242] [DEBUG] [app] Ending request: urn:request:381d52f5-dd27-4c45-9f32-2b86892aebd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:381d52f5-dd27-4c45-9f32-2b86892aebd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:06:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:06:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:37,636 [241] [DEBUG] [app] Starting request: urn:request:9ecf26e0-bdb9-45d1-86aa-8c48e8d41681 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:37,636 [239] [DEBUG] [app] Starting request: urn:request:d6c45412-d052-4b81-8686-0cda44fb52fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:37,636 [241] [DEBUG] [app] Ending request: urn:request:9ecf26e0-bdb9-45d1-86aa-8c48e8d41681 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ecf26e0-bdb9-45d1-86aa-8c48e8d41681', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:37,636 [239] [DEBUG] [app] Ending request: urn:request:d6c45412-d052-4b81-8686-0cda44fb52fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d6c45412-d052-4b81-8686-0cda44fb52fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:06:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:06:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:37,647 [244] [DEBUG] [app] Ending request: urn:request:1e70a056-576e-4625-a4a0-f2db4d051a11 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e70a056-576e-4625-a4a0-f2db4d051a11', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:37,647 [243] [DEBUG] [app] Ending request: urn:request:eb0e8408-4e6d-4533-91e8-d501a705fc3f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb0e8408-4e6d-4533-91e8-d501a705fc3f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:06:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:06:37,868 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:06:37,872 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:06:37,875 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:06:37,878 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:06:37,880 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:06:37,883 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:06:37,885 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:06:37,893 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:06:37,899 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:06:38,121 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:06:38,158 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:06:38,165 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:06:38,168 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:06:38,170 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:06:38,173 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityscanningnotificationworker stdout | 2023-01-17 05:06:38,240 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:06:38,247 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:06:38,252 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:06:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:43.935629+00:00 (in 5.200130 seconds) notificationworker stdout | 2023-01-17 05:06:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:48 UTC)" (scheduled at 2023-01-17 05:06:38.735066+00:00) notificationworker stdout | 2023-01-17 05:06:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 38, 735907), True, datetime.datetime(2023, 1, 17, 5, 6, 38, 735907), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:06:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:06:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:48.738458+00:00 (in 9.707237 seconds) repositorygcworker stdout | 2023-01-17 05:06:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:39 UTC)" (scheduled at 2023-01-17 05:06:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:06:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:06:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 39, 31608), True, datetime.datetime(2023, 1, 17, 5, 6, 39, 31608), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:06:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:06:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:06:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:06:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:52.837365+00:00 (in 13.606868 seconds) securityworker stdout | 2023-01-17 05:06:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:09 UTC)" (scheduled at 2023-01-17 05:06:39.230038+00:00) securityworker stdout | 2023-01-17 05:06:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:06:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:06:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:06:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:06:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:06:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:06:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:06:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:06:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:20.242198+00:00 (in 40.907901 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:06:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:39 UTC)" (scheduled at 2023-01-17 05:06:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:06:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 6, 39, 334718), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:06:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 39, 334718), True, datetime.datetime(2023, 1, 17, 5, 6, 39, 334718), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:06:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 39, 334718), True, datetime.datetime(2023, 1, 17, 5, 6, 39, 334718), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 6, 39, 334718), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:06:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:06:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:48.735066+00:00 (in 4.798938 seconds) notificationworker stdout | 2023-01-17 05:06:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:43 UTC)" (scheduled at 2023-01-17 05:06:43.935629+00:00) notificationworker stdout | 2023-01-17 05:06:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:06:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:06:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:06:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:48.830274+00:00 (in 4.188275 seconds) exportactionlogsworker stdout | 2023-01-17 05:06:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:44 UTC)" (scheduled at 2023-01-17 05:06:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:06:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 6, 44, 642380), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:06:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 44, 642380), True, datetime.datetime(2023, 1, 17, 5, 6, 44, 642380), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:06:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 44, 642380), True, datetime.datetime(2023, 1, 17, 5, 6, 44, 642380), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 6, 44, 642380), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:06:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:06:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:44 UTC)" executed successfully builder stdout | 2023-01-17 05:06:47,298 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:47,307 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:47,307 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:47,313 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:06:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:06:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:59.337413+00:00 (in 11.404464 seconds) namespacegcworker stdout | 2023-01-17 05:06:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:47 UTC)" (scheduled at 2023-01-17 05:06:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:06:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:06:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:58.735066+00:00 (in 9.999454 seconds) notificationworker stdout | 2023-01-17 05:06:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:58 UTC)" (scheduled at 2023-01-17 05:06:48.735066+00:00) notificationworker stdout | 2023-01-17 05:06:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 48, 736003), True, datetime.datetime(2023, 1, 17, 5, 6, 48, 736003), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:06:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:06:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:04.738062+00:00 (in 15.999168 seconds) repositorygcworker stdout | 2023-01-17 05:06:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:48 UTC)" (scheduled at 2023-01-17 05:06:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:06:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:06:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:06:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:06:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:24.641816+00:00 (in 35.811138 seconds) exportactionlogsworker stdout | 2023-01-17 05:06:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:48 UTC)" (scheduled at 2023-01-17 05:06:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:06:48,830 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:06:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 48, 831026), True, datetime.datetime(2023, 1, 17, 5, 6, 48, 831026), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:06:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:06:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:06:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:06:52,626 [239] [DEBUG] [app] Starting request: urn:request:aeb5581d-0008-4708-b4b8-535f71ca5bb8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:52,626 [244] [DEBUG] [app] Starting request: urn:request:29be6ca6-948a-45bb-a9ea-7502ce432116 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:06:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:52,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:52,631 [232] [DEBUG] [app] Starting request: urn:request:24b5699b-078d-4721-bc0f-70e931970e2a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,632 [240] [DEBUG] [app] Starting request: urn:request:a2344e11-dcc7-4590-9c13-6f82ba32f19a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,632 [240] [DEBUG] [app] Ending request: urn:request:a2344e11-dcc7-4590-9c13-6f82ba32f19a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2344e11-dcc7-4590-9c13-6f82ba32f19a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,632 [232] [DEBUG] [app] Ending request: urn:request:24b5699b-078d-4721-bc0f-70e931970e2a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24b5699b-078d-4721-bc0f-70e931970e2a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:06:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:06:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:52,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:52,637 [243] [DEBUG] [app] Starting request: urn:request:d4a5d73e-fadc-4198-a0cf-1ea8f5e7575b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:52,637 [244] [DEBUG] [app] Starting request: urn:request:c1d0a22b-a042-4b7f-b7cd-809435bb7d50 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:52,638 [243] [DEBUG] [app] Ending request: urn:request:d4a5d73e-fadc-4198-a0cf-1ea8f5e7575b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4a5d73e-fadc-4198-a0cf-1ea8f5e7575b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:52,638 [244] [DEBUG] [app] Ending request: urn:request:c1d0a22b-a042-4b7f-b7cd-809435bb7d50 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c1d0a22b-a042-4b7f-b7cd-809435bb7d50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:06:52,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:52,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:52,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:52,649 [244] [DEBUG] [app] Ending request: urn:request:29be6ca6-948a-45bb-a9ea-7502ce432116 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29be6ca6-948a-45bb-a9ea-7502ce432116', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:52,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:52,649 [239] [DEBUG] [app] Ending request: urn:request:aeb5581d-0008-4708-b4b8-535f71ca5bb8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aeb5581d-0008-4708-b4b8-535f71ca5bb8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 05:06:52,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:06:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:09.230038+00:00 (in 16.392210 seconds) securityworker stdout | 2023-01-17 05:06:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:22 UTC)" (scheduled at 2023-01-17 05:06:52.837365+00:00) securityworker stdout | 2023-01-17 05:06:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:06:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:06:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:27.226424+00:00 (in 29.999573 seconds) buildlogsarchiver stdout | 2023-01-17 05:06:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:27 UTC)" (scheduled at 2023-01-17 05:06:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:06:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 6, 57, 227191), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:06:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:06:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:06:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:27 UTC)" executed successfully builder stdout | 2023-01-17 05:06:57,323 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:57,331 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:57,331 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:57,337 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:08.735066+00:00 (in 9.999483 seconds) notificationworker stdout | 2023-01-17 05:06:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:08 UTC)" (scheduled at 2023-01-17 05:06:58.735066+00:00) notificationworker stdout | 2023-01-17 05:06:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 58, 736016), True, datetime.datetime(2023, 1, 17, 5, 6, 58, 736016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:06:59,157 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} namespacegcworker stdout | 2023-01-17 05:06:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:06:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:47.932490+00:00 (in 48.594639 seconds) namespacegcworker stdout | 2023-01-17 05:06:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:59 UTC)" (scheduled at 2023-01-17 05:06:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:06:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:06:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 59, 338212), True, datetime.datetime(2023, 1, 17, 5, 6, 59, 338212), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:06:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:06:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:06:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:59 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:06:59,427 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:06:59,702 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:07:00,434 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:07:00,471 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:07:01,675 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:07:02,517 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:07:02,573 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:07:02,992 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:07:03,748 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:07:03,966 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} repositorygcworker stdout | 2023-01-17 05:07:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:07:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:39.030814+00:00 (in 34.292278 seconds) repositorygcworker stdout | 2023-01-17 05:07:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:12:04 UTC)" (scheduled at 2023-01-17 05:07:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:07:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 7, 4, 738920), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:07:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 7, 4, 738920), True, datetime.datetime(2023, 1, 17, 5, 7, 4, 738920), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:07:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 7, 4, 738920), True, datetime.datetime(2023, 1, 17, 5, 7, 4, 738920), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 7, 4, 738920), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:07:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:07:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:12:04 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:07:04,856 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:07:05,479 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:07:05,567 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:07:05,571 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:07:05,573 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:07:05,649 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:07:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:07:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:35.848675+00:00 (in 29.999471 seconds) gcworker stdout | 2023-01-17 05:07:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:35 UTC)" (scheduled at 2023-01-17 05:07:05.848675+00:00) gcworker stdout | 2023-01-17 05:07:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722425849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:07:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:07:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:07:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:07:06,842 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:07:07,347 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:07,356 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:07,356 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:07,361 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:07,416 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:07:07,625 [243] [DEBUG] [app] Starting request: urn:request:aa85bdb2-b7ef-46c9-83bb-9410056a4854 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:07,625 [244] [DEBUG] [app] Starting request: urn:request:e566bab8-e56d-4e9e-a0ac-e0e0a15e3c26 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:07,630 [242] [DEBUG] [app] Starting request: urn:request:33ff8e11-5111-4074-9b49-15961a52fcb9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,631 [240] [DEBUG] [app] Starting request: urn:request:98eb61da-8591-4e34-81b8-d093f9e86ae1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,631 [242] [DEBUG] [app] Ending request: urn:request:33ff8e11-5111-4074-9b49-15961a52fcb9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:33ff8e11-5111-4074-9b49-15961a52fcb9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,631 [240] [DEBUG] [app] Ending request: urn:request:98eb61da-8591-4e34-81b8-d093f9e86ae1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:98eb61da-8591-4e34-81b8-d093f9e86ae1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:07:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:07:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:07,636 [243] [DEBUG] [app] Starting request: urn:request:63a3ce35-74b7-498f-b894-ed8741cff981 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:07,636 [243] [DEBUG] [app] Ending request: urn:request:63a3ce35-74b7-498f-b894-ed8741cff981 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:63a3ce35-74b7-498f-b894-ed8741cff981', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:07,637 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:07,638 [239] [DEBUG] [app] Starting request: urn:request:61272069-529f-4bd9-9d5e-98ee336dd4f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:07,639 [239] [DEBUG] [app] Ending request: urn:request:61272069-529f-4bd9-9d5e-98ee336dd4f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61272069-529f-4bd9-9d5e-98ee336dd4f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:07,639 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:07,639 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:07,640 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:07,644 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:07,647 [244] [DEBUG] [app] Ending request: urn:request:e566bab8-e56d-4e9e-a0ac-e0e0a15e3c26 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e566bab8-e56d-4e9e-a0ac-e0e0a15e3c26', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:07:07,649 [243] [DEBUG] [app] Ending request: urn:request:aa85bdb2-b7ef-46c9-83bb-9410056a4854 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aa85bdb2-b7ef-46c9-83bb-9410056a4854', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:07,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:07,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.025) gunicorn-registry stdout | 2023-01-17 05:07:07,876 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:07:07,879 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:07:07,882 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:07:07,885 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:07:07,887 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:07:07,890 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:07:07,892 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:07:07,899 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:07:07,907 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gcworker stdout | 2023-01-17 05:07:08,143 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:07:08,164 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:07:08,170 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:07:08,174 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:07:08,177 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:07:08,180 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:07:08,258 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:07:08,260 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:07:08,275 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:07:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:18.735066+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:07:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:18 UTC)" (scheduled at 2023-01-17 05:07:08.735066+00:00) notificationworker stdout | 2023-01-17 05:07:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 8, 735960), True, datetime.datetime(2023, 1, 17, 5, 7, 8, 735960), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:07:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:22.837365+00:00 (in 13.606863 seconds) securityworker stdout | 2023-01-17 05:07:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:39 UTC)" (scheduled at 2023-01-17 05:07:09.230038+00:00) securityworker stdout | 2023-01-17 05:07:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:07:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:07:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:07:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:39 UTC)" executed successfully builder stdout | 2023-01-17 05:07:17,372 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:17,380 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:17,380 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:17,386 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:28.735066+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:07:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:28 UTC)" (scheduled at 2023-01-17 05:07:18.735066+00:00) notificationworker stdout | 2023-01-17 05:07:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 18, 735937), True, datetime.datetime(2023, 1, 17, 5, 7, 18, 735937), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:07:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:07:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:25.334138+00:00 (in 5.091350 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:07:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:20 UTC)" (scheduled at 2023-01-17 05:07:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:07:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:07:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 20, 243169), True, datetime.datetime(2023, 1, 17, 5, 7, 20, 243169), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:07:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:07:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:07:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:07:22,625 [239] [DEBUG] [app] Starting request: urn:request:2f004782-05ea-4360-8fa5-19d75ec210b9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:22,625 [244] [DEBUG] [app] Starting request: urn:request:133362ae-1e3b-490a-9633-9dbe12e02d27 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:22,630 [232] [DEBUG] [app] Starting request: urn:request:688651b5-2016-4f06-92ab-ab437935d5e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:22,630 [240] [DEBUG] [app] Starting request: urn:request:7d382eb3-d6de-42ab-9858-510f90cb105d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:22,630 [232] [DEBUG] [app] Ending request: urn:request:688651b5-2016-4f06-92ab-ab437935d5e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:688651b5-2016-4f06-92ab-ab437935d5e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:07:22,631 [240] [DEBUG] [app] Ending request: urn:request:7d382eb3-d6de-42ab-9858-510f90cb105d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d382eb3-d6de-42ab-9858-510f90cb105d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:07:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:07:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:22,636 [243] [DEBUG] [app] Starting request: urn:request:3e61bcbc-9a12-4cbd-899e-eb166247f156 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:22,636 [244] [DEBUG] [app] Starting request: urn:request:899cf9bf-698f-40f3-a0b5-892afb2e9540 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:22,636 [243] [DEBUG] [app] Ending request: urn:request:3e61bcbc-9a12-4cbd-899e-eb166247f156 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e61bcbc-9a12-4cbd-899e-eb166247f156', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:22,637 [244] [DEBUG] [app] Ending request: urn:request:899cf9bf-698f-40f3-a0b5-892afb2e9540 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:899cf9bf-698f-40f3-a0b5-892afb2e9540', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:22,648 [244] [DEBUG] [app] Ending request: urn:request:133362ae-1e3b-490a-9633-9dbe12e02d27 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:133362ae-1e3b-490a-9633-9dbe12e02d27', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:22,648 [239] [DEBUG] [app] Ending request: urn:request:2f004782-05ea-4360-8fa5-19d75ec210b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f004782-05ea-4360-8fa5-19d75ec210b9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:07:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:07:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:39.230038+00:00 (in 16.392209 seconds) securityworker stdout | 2023-01-17 05:07:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:52 UTC)" (scheduled at 2023-01-17 05:07:22.837365+00:00) securityworker stdout | 2023-01-17 05:07:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:07:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:07:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:48.830274+00:00 (in 24.188048 seconds) exportactionlogsworker stdout | 2023-01-17 05:07:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:24 UTC)" (scheduled at 2023-01-17 05:07:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:07:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:07:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:07:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:07:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:20.242198+00:00 (in 54.907613 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:07:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:25 UTC)" (scheduled at 2023-01-17 05:07:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:07:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:07:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:07:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:07:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:57.226424+00:00 (in 29.999541 seconds) buildlogsarchiver stdout | 2023-01-17 05:07:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:57 UTC)" (scheduled at 2023-01-17 05:07:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:07:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 7, 27, 227246), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:07:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:07:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:07:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:57 UTC)" executed successfully builder stdout | 2023-01-17 05:07:27,396 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:27,404 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:27,404 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:27,410 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:38.735066+00:00 (in 9.999508 seconds) notificationworker stdout | 2023-01-17 05:07:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:38 UTC)" (scheduled at 2023-01-17 05:07:28.735066+00:00) notificationworker stdout | 2023-01-17 05:07:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 28, 736016), True, datetime.datetime(2023, 1, 17, 5, 7, 28, 736016), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:07:29,192 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:07:29,438 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:07:29,722 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:07:30,443 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:07:30,482 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:07:31,708 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:07:32,553 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:07:32,579 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:07:32,997 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:07:33,782 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:07:33,981 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:07:34,891 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:07:35,514 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:07:35,573 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:07:35,577 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:07:35,580 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:07:35,685 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:07:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:07:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:05.848675+00:00 (in 29.999506 seconds) gcworker stdout | 2023-01-17 05:07:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:05 UTC)" (scheduled at 2023-01-17 05:07:35.848675+00:00) gcworker stdout | 2023-01-17 05:07:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722455849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:07:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:07:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:07:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:07:36,854 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} builder stdout | 2023-01-17 05:07:37,420 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:37,428 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:37,428 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:37,434 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:37,451 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:07:37,626 [239] [DEBUG] [app] Starting request: urn:request:21eb1289-c43f-4fb9-8f08-67a4fcc3446e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:37,626 [243] [DEBUG] [app] Starting request: urn:request:aec594e3-b55e-46e5-967b-0513ad725093 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:37,631 [242] [DEBUG] [app] Starting request: urn:request:80f2a103-3ac8-4c87-a13f-3bdb737da829 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:37,631 [232] [DEBUG] [app] Starting request: urn:request:9a0fc5ac-a970-49f5-890a-301a3ac72966 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:37,631 [242] [DEBUG] [app] Ending request: urn:request:80f2a103-3ac8-4c87-a13f-3bdb737da829 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:80f2a103-3ac8-4c87-a13f-3bdb737da829', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:07:37,632 [232] [DEBUG] [app] Ending request: urn:request:9a0fc5ac-a970-49f5-890a-301a3ac72966 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a0fc5ac-a970-49f5-890a-301a3ac72966', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:07:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:37,636 [244] [DEBUG] [app] Starting request: urn:request:10907956-86d4-45ec-82a5-e441b0cd1cec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:37,636 [243] [DEBUG] [app] Starting request: urn:request:99a1f98f-611a-4137-87a4-63080065703d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:37,636 [244] [DEBUG] [app] Ending request: urn:request:10907956-86d4-45ec-82a5-e441b0cd1cec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10907956-86d4-45ec-82a5-e441b0cd1cec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:37,637 [243] [DEBUG] [app] Ending request: urn:request:99a1f98f-611a-4137-87a4-63080065703d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:99a1f98f-611a-4137-87a4-63080065703d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:07:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:37,647 [243] [DEBUG] [app] Ending request: urn:request:aec594e3-b55e-46e5-967b-0513ad725093 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aec594e3-b55e-46e5-967b-0513ad725093', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:37,647 [239] [DEBUG] [app] Ending request: urn:request:21eb1289-c43f-4fb9-8f08-67a4fcc3446e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21eb1289-c43f-4fb9-8f08-67a4fcc3446e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:07:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:07:37,883 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:07:37,886 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:07:37,888 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:07:37,892 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:07:37,895 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:07:37,897 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:07:37,899 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:07:37,906 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:07:37,914 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:07:38,171 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gcworker stdout | 2023-01-17 05:07:38,173 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:07:38,176 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:07:38,181 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:07:38,183 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:07:38,186 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} blobuploadcleanupworker stdout | 2023-01-17 05:07:38,292 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:07:38,295 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:07:38,309 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:07:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:43.935629+00:00 (in 5.200104 seconds) notificationworker stdout | 2023-01-17 05:07:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:48 UTC)" (scheduled at 2023-01-17 05:07:38.735066+00:00) notificationworker stdout | 2023-01-17 05:07:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 38, 735928), True, datetime.datetime(2023, 1, 17, 5, 7, 38, 735928), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:07:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:07:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:48.738458+00:00 (in 9.707166 seconds) repositorygcworker stdout | 2023-01-17 05:07:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:39 UTC)" (scheduled at 2023-01-17 05:07:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:07:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:07:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 39, 31688), True, datetime.datetime(2023, 1, 17, 5, 7, 39, 31688), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:07:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:07:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:07:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:07:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:52.837365+00:00 (in 13.606897 seconds) securityworker stdout | 2023-01-17 05:07:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:09 UTC)" (scheduled at 2023-01-17 05:07:39.230038+00:00) securityworker stdout | 2023-01-17 05:07:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:07:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:07:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:07:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:07:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:07:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:07:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:07:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:48.735066+00:00 (in 4.798992 seconds) notificationworker stdout | 2023-01-17 05:07:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:43 UTC)" (scheduled at 2023-01-17 05:07:43.935629+00:00) notificationworker stdout | 2023-01-17 05:07:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:07:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:43 UTC)" executed successfully builder stdout | 2023-01-17 05:07:47,444 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:47,453 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:47,453 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:47,459 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:07:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:07:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:59.337413+00:00 (in 11.404464 seconds) namespacegcworker stdout | 2023-01-17 05:07:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:47 UTC)" (scheduled at 2023-01-17 05:07:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:07:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:07:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:07:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:58.735066+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:07:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:58 UTC)" (scheduled at 2023-01-17 05:07:48.735066+00:00) notificationworker stdout | 2023-01-17 05:07:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 48, 735954), True, datetime.datetime(2023, 1, 17, 5, 7, 48, 735954), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:07:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:07:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:39.030814+00:00 (in 50.291892 seconds) repositorygcworker stdout | 2023-01-17 05:07:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:48 UTC)" (scheduled at 2023-01-17 05:07:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:07:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:07:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:07:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:07:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:07:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:24.641816+00:00 (in 35.811093 seconds) exportactionlogsworker stdout | 2023-01-17 05:07:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:48 UTC)" (scheduled at 2023-01-17 05:07:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:07:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:07:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 48, 831139), True, datetime.datetime(2023, 1, 17, 5, 7, 48, 831139), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:07:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:07:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:07:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:07:52,625 [244] [DEBUG] [app] Starting request: urn:request:aa5e2c80-974b-4ef5-bc78-d1efe6e2c6f8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,625 [243] [DEBUG] [app] Starting request: urn:request:22186a02-56ad-4829-bc1e-d12268ec902f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:52,630 [242] [DEBUG] [app] Starting request: urn:request:f3b6eca3-350d-489b-86fd-906ec49af096 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:52,631 [240] [DEBUG] [app] Starting request: urn:request:1216f38c-c223-4e04-abd7-c5b45da3cadb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:52,631 [242] [DEBUG] [app] Ending request: urn:request:f3b6eca3-350d-489b-86fd-906ec49af096 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3b6eca3-350d-489b-86fd-906ec49af096', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:52,631 [240] [DEBUG] [app] Ending request: urn:request:1216f38c-c223-4e04-abd7-c5b45da3cadb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1216f38c-c223-4e04-abd7-c5b45da3cadb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:07:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:07:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:52,635 [241] [DEBUG] [app] Starting request: urn:request:f517ff19-fc18-4023-8dc5-e28b6b9731a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,636 [239] [DEBUG] [app] Starting request: urn:request:f2626815-8829-481f-94cf-e5430d6b1300 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,636 [241] [DEBUG] [app] Ending request: urn:request:f517ff19-fc18-4023-8dc5-e28b6b9731a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f517ff19-fc18-4023-8dc5-e28b6b9731a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:52,636 [239] [DEBUG] [app] Ending request: urn:request:f2626815-8829-481f-94cf-e5430d6b1300 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2626815-8829-481f-94cf-e5430d6b1300', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:07:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:52,647 [244] [DEBUG] [app] Ending request: urn:request:aa5e2c80-974b-4ef5-bc78-d1efe6e2c6f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aa5e2c80-974b-4ef5-bc78-d1efe6e2c6f8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:52,647 [243] [DEBUG] [app] Ending request: urn:request:22186a02-56ad-4829-bc1e-d12268ec902f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:22186a02-56ad-4829-bc1e-d12268ec902f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:07:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:07:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:09.230038+00:00 (in 16.392211 seconds) securityworker stdout | 2023-01-17 05:07:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:22 UTC)" (scheduled at 2023-01-17 05:07:52.837365+00:00) securityworker stdout | 2023-01-17 05:07:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:07:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:07:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:27.226424+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2023-01-17 05:07:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:27 UTC)" (scheduled at 2023-01-17 05:07:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:07:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 7, 57, 227236), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:07:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:07:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:07:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:27 UTC)" executed successfully builder stdout | 2023-01-17 05:07:57,465 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:57,473 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:57,473 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:57,479 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:08.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 05:07:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:08 UTC)" (scheduled at 2023-01-17 05:07:58.735066+00:00) notificationworker stdout | 2023-01-17 05:07:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 58, 735974), True, datetime.datetime(2023, 1, 17, 5, 7, 58, 735974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:07:59,208 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} namespacegcworker stdout | 2023-01-17 05:07:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:07:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:47.932490+00:00 (in 48.594598 seconds) namespacegcworker stdout | 2023-01-17 05:07:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:59 UTC)" (scheduled at 2023-01-17 05:07:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:07:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:07:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 59, 338357), True, datetime.datetime(2023, 1, 17, 5, 7, 59, 338357), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:07:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:07:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:07:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:59 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:07:59,474 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:07:59,757 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:08:00,478 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:08:00,503 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:08:01,720 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:08:02,566 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:08:02,587 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:08:03,025 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:08:03,817 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:08:03,989 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:08:04,912 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:08:05,550 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:08:05,579 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:08:05,584 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:08:05,586 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:08:05,714 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:08:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:08:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:35.848675+00:00 (in 29.999520 seconds) gcworker stdout | 2023-01-17 05:08:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:35 UTC)" (scheduled at 2023-01-17 05:08:05.848675+00:00) gcworker stdout | 2023-01-17 05:08:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722485849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:08:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:08:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:08:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:08:06,871 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:08:07,481 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} builder stdout | 2023-01-17 05:08:07,489 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:07,497 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:07,497 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:07,503 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:07,626 [244] [DEBUG] [app] Starting request: urn:request:61aef844-186f-411d-8987-bbb80c303f46 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:07,626 [243] [DEBUG] [app] Starting request: urn:request:c44f2039-fc13-407c-9f58-23c4d53e569d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:07,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,628 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:07,631 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:07,631 [232] [DEBUG] [app] Starting request: urn:request:c941e096-c9ce-41dd-bc76-02feea0674ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,632 [236] [DEBUG] [app] Starting request: urn:request:c994b2a5-20dc-4d3d-a5e0-02d379574839 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,632 [232] [DEBUG] [app] Ending request: urn:request:c941e096-c9ce-41dd-bc76-02feea0674ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c941e096-c9ce-41dd-bc76-02feea0674ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,632 [236] [DEBUG] [app] Ending request: urn:request:c994b2a5-20dc-4d3d-a5e0-02d379574839 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c994b2a5-20dc-4d3d-a5e0-02d379574839', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:08:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:08:07,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:07,633 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:07,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:07,637 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:07,637 [243] [DEBUG] [app] Starting request: urn:request:413f5479-0d81-4ec9-b14e-07788be4ad14 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:07,637 [243] [DEBUG] [app] Ending request: urn:request:413f5479-0d81-4ec9-b14e-07788be4ad14 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:413f5479-0d81-4ec9-b14e-07788be4ad14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:07,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:07,638 [241] [DEBUG] [app] Starting request: urn:request:cfd89142-6017-4e3e-ac18-59bc0101b60b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:07,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:07,638 [241] [DEBUG] [app] Ending request: urn:request:cfd89142-6017-4e3e-ac18-59bc0101b60b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cfd89142-6017-4e3e-ac18-59bc0101b60b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:08:07,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:07,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:07,648 [243] [DEBUG] [app] Ending request: urn:request:c44f2039-fc13-407c-9f58-23c4d53e569d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c44f2039-fc13-407c-9f58-23c4d53e569d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:07,648 [244] [DEBUG] [app] Ending request: urn:request:61aef844-186f-411d-8987-bbb80c303f46 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61aef844-186f-411d-8987-bbb80c303f46', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:08:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:08:07,892 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:08:07,894 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:08:07,897 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:08:07,899 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:08:07,902 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:08:07,904 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:08:07,906 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:08:07,913 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:08:07,921 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:08:08,178 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gcworker stdout | 2023-01-17 05:08:08,180 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 05:08:08,182 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:08:08,186 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:08:08,189 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:08:08,192 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:08:08,302 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:08:08,314 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:08:08,343 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:08:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:18.735066+00:00 (in 9.999586 seconds) notificationworker stdout | 2023-01-17 05:08:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:18 UTC)" (scheduled at 2023-01-17 05:08:08.735066+00:00) notificationworker stdout | 2023-01-17 05:08:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 8, 735877), True, datetime.datetime(2023, 1, 17, 5, 8, 8, 735877), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:08:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:22.837365+00:00 (in 13.606862 seconds) securityworker stdout | 2023-01-17 05:08:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:39 UTC)" (scheduled at 2023-01-17 05:08:09.230038+00:00) securityworker stdout | 2023-01-17 05:08:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:08:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:08:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:08:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:08:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:08:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:08:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:39 UTC)" executed successfully builder stdout | 2023-01-17 05:08:17,514 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:17,526 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:17,526 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:17,533 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:28.735066+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 05:08:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:28 UTC)" (scheduled at 2023-01-17 05:08:18.735066+00:00) notificationworker stdout | 2023-01-17 05:08:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 18, 735934), True, datetime.datetime(2023, 1, 17, 5, 8, 18, 735934), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:08:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:08:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:25.334138+00:00 (in 5.091476 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:08:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:20 UTC)" (scheduled at 2023-01-17 05:08:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:08:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:08:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 20, 243114), True, datetime.datetime(2023, 1, 17, 5, 8, 20, 243114), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:08:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:08:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:08:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:08:22,625 [244] [DEBUG] [app] Starting request: urn:request:4662090d-0d4d-4715-85fa-a1f53760493f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:22,625 [243] [DEBUG] [app] Starting request: urn:request:c9d2259f-88f8-4e3e-a085-fc42eaf3aa6b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:22,630 [242] [DEBUG] [app] Starting request: urn:request:a1f948d7-32e7-4dfb-8923-1998d79d17ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:22,630 [232] [DEBUG] [app] Starting request: urn:request:3107379f-8885-42ad-aee8-f6e9647c964c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:22,631 [242] [DEBUG] [app] Ending request: urn:request:a1f948d7-32e7-4dfb-8923-1998d79d17ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1f948d7-32e7-4dfb-8923-1998d79d17ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:08:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:08:22,631 [232] [DEBUG] [app] Ending request: urn:request:3107379f-8885-42ad-aee8-f6e9647c964c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3107379f-8885-42ad-aee8-f6e9647c964c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:08:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:08:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:22,636 [241] [DEBUG] [app] Starting request: urn:request:dd42f74a-3514-4ac5-8da1-1b3f3f1be28e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:22,636 [239] [DEBUG] [app] Starting request: urn:request:2300789a-8466-4378-9afd-b300c20acdbb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:22,636 [241] [DEBUG] [app] Ending request: urn:request:dd42f74a-3514-4ac5-8da1-1b3f3f1be28e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd42f74a-3514-4ac5-8da1-1b3f3f1be28e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:22,637 [239] [DEBUG] [app] Ending request: urn:request:2300789a-8466-4378-9afd-b300c20acdbb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2300789a-8466-4378-9afd-b300c20acdbb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:08:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:22,648 [244] [DEBUG] [app] Ending request: urn:request:4662090d-0d4d-4715-85fa-a1f53760493f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4662090d-0d4d-4715-85fa-a1f53760493f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:22,648 [243] [DEBUG] [app] Ending request: urn:request:c9d2259f-88f8-4e3e-a085-fc42eaf3aa6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9d2259f-88f8-4e3e-a085-fc42eaf3aa6b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:08:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:08:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:39.230038+00:00 (in 16.392187 seconds) securityworker stdout | 2023-01-17 05:08:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:52 UTC)" (scheduled at 2023-01-17 05:08:22.837365+00:00) securityworker stdout | 2023-01-17 05:08:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:08:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:08:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:48.830274+00:00 (in 24.187997 seconds) exportactionlogsworker stdout | 2023-01-17 05:08:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:24 UTC)" (scheduled at 2023-01-17 05:08:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:08:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:08:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:08:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:08:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:20.242198+00:00 (in 54.907542 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:08:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:25 UTC)" (scheduled at 2023-01-17 05:08:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:08:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:08:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:08:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:08:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:57.226424+00:00 (in 29.999518 seconds) buildlogsarchiver stdout | 2023-01-17 05:08:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:57 UTC)" (scheduled at 2023-01-17 05:08:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:08:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 8, 27, 227292), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:08:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:08:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:08:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:57 UTC)" executed successfully builder stdout | 2023-01-17 05:08:27,539 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:27,548 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:27,548 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:27,553 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:38.735066+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:08:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:38 UTC)" (scheduled at 2023-01-17 05:08:28.735066+00:00) notificationworker stdout | 2023-01-17 05:08:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 28, 736008), True, datetime.datetime(2023, 1, 17, 5, 8, 28, 736008), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:08:29,244 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:08:29,483 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:08:29,785 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:08:30,514 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:08:30,538 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:08:31,755 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:08:32,602 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:08:32,605 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:08:33,044 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:08:33,830 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:08:33,995 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:08:34,943 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:08:35,586 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:08:35,589 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:08:35,591 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:08:35,593 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:08:35,750 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:08:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:08:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:05.848675+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 05:08:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:05 UTC)" (scheduled at 2023-01-17 05:08:35.848675+00:00) gcworker stdout | 2023-01-17 05:08:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722515849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:08:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:08:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:08:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:08:36,901 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:08:37,489 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} builder stdout | 2023-01-17 05:08:37,563 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:37,572 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:37,572 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:37,578 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:37,626 [244] [DEBUG] [app] Starting request: urn:request:ebd24cf3-01c0-40c5-bef0-8d17395279f8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:37,626 [243] [DEBUG] [app] Starting request: urn:request:540acee0-2974-4df1-b754-e69bd7219baa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:37,631 [232] [DEBUG] [app] Starting request: urn:request:2c7469ac-04ad-4d3a-b748-3afbd7cdb199 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:37,631 [234] [DEBUG] [app] Starting request: urn:request:867b4f3c-e833-4adb-b2b5-591249737eb1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:37,632 [232] [DEBUG] [app] Ending request: urn:request:2c7469ac-04ad-4d3a-b748-3afbd7cdb199 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c7469ac-04ad-4d3a-b748-3afbd7cdb199', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:08:37,632 [234] [DEBUG] [app] Ending request: urn:request:867b4f3c-e833-4adb-b2b5-591249737eb1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:867b4f3c-e833-4adb-b2b5-591249737eb1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:08:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:08:37,632 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:08:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:37,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:37,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:37,637 [239] [DEBUG] [app] Starting request: urn:request:723b6226-991f-4a46-88aa-84d55a47f9d6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:37,637 [243] [DEBUG] [app] Starting request: urn:request:1fee7c2b-5d06-4787-b951-db0d9cca759c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:37,637 [239] [DEBUG] [app] Ending request: urn:request:723b6226-991f-4a46-88aa-84d55a47f9d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:723b6226-991f-4a46-88aa-84d55a47f9d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:37,637 [243] [DEBUG] [app] Ending request: urn:request:1fee7c2b-5d06-4787-b951-db0d9cca759c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fee7c2b-5d06-4787-b951-db0d9cca759c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:37,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:37,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:37,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:37,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:37,648 [244] [DEBUG] [app] Ending request: urn:request:ebd24cf3-01c0-40c5-bef0-8d17395279f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ebd24cf3-01c0-40c5-bef0-8d17395279f8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:37,648 [243] [DEBUG] [app] Ending request: urn:request:540acee0-2974-4df1-b754-e69bd7219baa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:540acee0-2974-4df1-b754-e69bd7219baa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:37,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:37,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:37,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:08:37,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-registry stdout | 2023-01-17 05:08:37,901 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:08:37,903 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:08:37,906 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:08:37,908 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:08:37,910 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:08:37,912 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:08:37,915 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:08:37,920 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:08:37,927 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:08:38,183 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:08:38,189 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:08:38,192 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:08:38,195 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:08:38,198 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:08:38,215 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:08:38,308 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:08:38,332 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:08:38,378 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:08:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:43.935629+00:00 (in 5.200104 seconds) notificationworker stdout | 2023-01-17 05:08:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:48 UTC)" (scheduled at 2023-01-17 05:08:38.735066+00:00) notificationworker stdout | 2023-01-17 05:08:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 38, 735921), True, datetime.datetime(2023, 1, 17, 5, 8, 38, 735921), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:08:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:08:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:48.738458+00:00 (in 9.707243 seconds) repositorygcworker stdout | 2023-01-17 05:08:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:39 UTC)" (scheduled at 2023-01-17 05:08:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:08:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:08:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 39, 31668), True, datetime.datetime(2023, 1, 17, 5, 8, 39, 31668), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:08:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:08:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:08:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:08:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:52.837365+00:00 (in 13.606895 seconds) securityworker stdout | 2023-01-17 05:08:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:09 UTC)" (scheduled at 2023-01-17 05:08:39.230038+00:00) securityworker stdout | 2023-01-17 05:08:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:08:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:08:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:08:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:08:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:48.735066+00:00 (in 4.798928 seconds) notificationworker stdout | 2023-01-17 05:08:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:43 UTC)" (scheduled at 2023-01-17 05:08:43.935629+00:00) notificationworker stdout | 2023-01-17 05:08:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:08:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:43 UTC)" executed successfully builder stdout | 2023-01-17 05:08:47,588 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:47,597 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:47,597 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:47,603 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:08:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:08:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:59.337413+00:00 (in 11.404424 seconds) namespacegcworker stdout | 2023-01-17 05:08:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:47 UTC)" (scheduled at 2023-01-17 05:08:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:08:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:08:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:08:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:56.935055+00:00 (in 8.199388 seconds) notificationworker stdout | 2023-01-17 05:08:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:58 UTC)" (scheduled at 2023-01-17 05:08:48.735066+00:00) notificationworker stdout | 2023-01-17 05:08:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 48, 736093), True, datetime.datetime(2023, 1, 17, 5, 8, 48, 736093), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:08:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:08:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:39.030814+00:00 (in 50.291870 seconds) repositorygcworker stdout | 2023-01-17 05:08:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:48 UTC)" (scheduled at 2023-01-17 05:08:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:08:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:08:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:08:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:08:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:08:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:24.641816+00:00 (in 35.811115 seconds) exportactionlogsworker stdout | 2023-01-17 05:08:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:48 UTC)" (scheduled at 2023-01-17 05:08:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:08:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:08:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 48, 831047), True, datetime.datetime(2023, 1, 17, 5, 8, 48, 831047), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:08:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:08:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:08:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:08:52,626 [244] [DEBUG] [app] Starting request: urn:request:e505eca0-bc4d-43fe-a374-918e69eaeca4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:52,626 [239] [DEBUG] [app] Starting request: urn:request:e5ed26f4-89c4-4c2e-9a47-37e13c9fcc9f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:08:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:52,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:52,631 [237] [DEBUG] [app] Starting request: urn:request:3a2c6457-2d57-443b-b2c1-a585a0d53c2d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,631 [242] [DEBUG] [app] Starting request: urn:request:c27e9266-0194-43a9-ae71-229ab10cf696 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,632 [237] [DEBUG] [app] Ending request: urn:request:3a2c6457-2d57-443b-b2c1-a585a0d53c2d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a2c6457-2d57-443b-b2c1-a585a0d53c2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:08:52,632 [242] [DEBUG] [app] Ending request: urn:request:c27e9266-0194-43a9-ae71-229ab10cf696 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c27e9266-0194-43a9-ae71-229ab10cf696', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) gunicorn-web stdout | 2023-01-17 05:08:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:08:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.003) gunicorn-web stdout | 2023-01-17 05:08:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:52,637 [241] [DEBUG] [app] Starting request: urn:request:9b7c42bc-466d-41a4-ab84-c2a2b7ef76aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:52,637 [243] [DEBUG] [app] Starting request: urn:request:8c917a0f-024e-4939-a972-6b8fcaf93f9e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:52,637 [241] [DEBUG] [app] Ending request: urn:request:9b7c42bc-466d-41a4-ab84-c2a2b7ef76aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b7c42bc-466d-41a4-ab84-c2a2b7ef76aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:52,638 [243] [DEBUG] [app] Ending request: urn:request:8c917a0f-024e-4939-a972-6b8fcaf93f9e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c917a0f-024e-4939-a972-6b8fcaf93f9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:08:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.003) gunicorn-web stdout | 2023-01-17 05:08:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:52,649 [244] [DEBUG] [app] Ending request: urn:request:e505eca0-bc4d-43fe-a374-918e69eaeca4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e505eca0-bc4d-43fe-a374-918e69eaeca4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:52,649 [239] [DEBUG] [app] Ending request: urn:request:e5ed26f4-89c4-4c2e-9a47-37e13c9fcc9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5ed26f4-89c4-4c2e-9a47-37e13c9fcc9f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:52,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:08:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:08:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:09.230038+00:00 (in 16.392203 seconds) securityworker stdout | 2023-01-17 05:08:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:22 UTC)" (scheduled at 2023-01-17 05:08:52.837365+00:00) securityworker stdout | 2023-01-17 05:08:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:08:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:58.735066+00:00 (in 1.799537 seconds) notificationworker stdout | 2023-01-17 05:08:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:13:56 UTC)" (scheduled at 2023-01-17 05:08:56.935055+00:00) notificationworker stdout | 2023-01-17 05:08:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 8, 56, 935896), 'notification/%']) notificationworker stdout | 2023-01-17 05:08:56,948 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 8, 56, 935896), True, datetime.datetime(2023, 1, 17, 5, 8, 56, 935896), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:08:56,950 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 8, 56, 935896), True, datetime.datetime(2023, 1, 17, 5, 8, 56, 935896), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 8, 56, 935896), 'notification/%']) notificationworker stdout | 2023-01-17 05:08:56,952 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:56,952 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:13:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:08:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:08:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:27.226424+00:00 (in 29.999537 seconds) buildlogsarchiver stdout | 2023-01-17 05:08:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:27 UTC)" (scheduled at 2023-01-17 05:08:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:08:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 8, 57, 227238), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:08:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:08:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:08:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:27 UTC)" executed successfully builder stdout | 2023-01-17 05:08:57,613 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:57,622 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:57,622 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:57,628 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:08.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:08:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:08 UTC)" (scheduled at 2023-01-17 05:08:58.735066+00:00) notificationworker stdout | 2023-01-17 05:08:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 58, 735995), True, datetime.datetime(2023, 1, 17, 5, 8, 58, 735995), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:08 UTC)" executed successfully securityworker stdout | 2023-01-17 05:08:59,276 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} namespacegcworker stdout | 2023-01-17 05:08:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:08:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:15.931994+00:00 (in 16.594120 seconds) namespacegcworker stdout | 2023-01-17 05:08:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:59 UTC)" (scheduled at 2023-01-17 05:08:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:08:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:08:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 59, 338285), True, datetime.datetime(2023, 1, 17, 5, 8, 59, 338285), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:08:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:08:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:08:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:59 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:08:59,516 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:08:59,797 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:09:00,550 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:09:00,568 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:09:01,790 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:09:02,638 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:09:02,641 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:09:03,050 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:09:03,866 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:09:04,030 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:09:04,971 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:09:05,596 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} logrotateworker stdout | 2023-01-17 05:09:05,598 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-secscan stdout | 2023-01-17 05:09:05,601 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:09:05,603 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:09:05,757 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:09:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:09:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:35.848675+00:00 (in 29.999488 seconds) gcworker stdout | 2023-01-17 05:09:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:35 UTC)" (scheduled at 2023-01-17 05:09:05.848675+00:00) gcworker stdout | 2023-01-17 05:09:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722545849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:09:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:09:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:09:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:09:06,931 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:09:07,496 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:09:07,625 [243] [DEBUG] [app] Starting request: urn:request:1225f17f-8491-412c-8d89-9c41d1307a1b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:07,625 [244] [DEBUG] [app] Starting request: urn:request:1a31b8dd-669c-4e85-b945-6933f22f49ca (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:07,630 [240] [DEBUG] [app] Starting request: urn:request:f72785d6-d4bd-4efb-9191-b00ed2665f6f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,631 [242] [DEBUG] [app] Starting request: urn:request:4e54ce0e-0f9b-448e-83d9-e051f32091b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,631 [240] [DEBUG] [app] Ending request: urn:request:f72785d6-d4bd-4efb-9191-b00ed2665f6f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f72785d6-d4bd-4efb-9191-b00ed2665f6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,631 [242] [DEBUG] [app] Ending request: urn:request:4e54ce0e-0f9b-448e-83d9-e051f32091b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e54ce0e-0f9b-448e-83d9-e051f32091b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:09:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:09:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:09:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:07,636 [244] [DEBUG] [app] Starting request: urn:request:a99577f3-220a-4ffb-bf2c-acf929251b43 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:07,636 [244] [DEBUG] [app] Ending request: urn:request:a99577f3-220a-4ffb-bf2c-acf929251b43 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a99577f3-220a-4ffb-bf2c-acf929251b43', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:07,636 [241] [DEBUG] [app] Starting request: urn:request:daebb04e-d87e-4168-aa74-d2d10809b9f9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:07,637 [241] [DEBUG] [app] Ending request: urn:request:daebb04e-d87e-4168-aa74-d2d10809b9f9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:daebb04e-d87e-4168-aa74-d2d10809b9f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:09:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql builder stdout | 2023-01-17 05:09:07,638 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-web stdout | 2023-01-17 05:09:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) builder stdout | 2023-01-17 05:09:07,646 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:07,646 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-web stdout | 2023-01-17 05:09:07,647 [244] [DEBUG] [app] Ending request: urn:request:1a31b8dd-669c-4e85-b945-6933f22f49ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1a31b8dd-669c-4e85-b945-6933f22f49ca', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:09:07,648 [243] [DEBUG] [app] Ending request: urn:request:1225f17f-8491-412c-8d89-9c41d1307a1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1225f17f-8491-412c-8d89-9c41d1307a1b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) builder stdout | 2023-01-17 05:09:07,652 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:09:07,910 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:09:07,912 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:09:07,915 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:09:07,917 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:09:07,919 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:09:07,922 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:09:07,925 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:09:07,927 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:09:07,932 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:09:08,189 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:09:08,196 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:09:08,198 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:09:08,201 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:09:08,204 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:09:08,249 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:09:08,319 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:09:08,357 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:09:08,408 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:09:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:18.735066+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:09:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:18 UTC)" (scheduled at 2023-01-17 05:09:08.735066+00:00) notificationworker stdout | 2023-01-17 05:09:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 8, 735939), True, datetime.datetime(2023, 1, 17, 5, 9, 8, 735939), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:09:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:22.837365+00:00 (in 13.606865 seconds) securityworker stdout | 2023-01-17 05:09:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:39 UTC)" (scheduled at 2023-01-17 05:09:09.230038+00:00) securityworker stdout | 2023-01-17 05:09:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:09:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:09:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:09,232 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:09,232 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:09,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:09:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:39 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:09:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:09:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:47.932490+00:00 (in 32.000044 seconds) namespacegcworker stdout | 2023-01-17 05:09:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:14:15 UTC)" (scheduled at 2023-01-17 05:09:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:09:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 9, 15, 932844), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:09:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 9, 15, 932844), True, datetime.datetime(2023, 1, 17, 5, 9, 15, 932844), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:09:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 9, 15, 932844), True, datetime.datetime(2023, 1, 17, 5, 9, 15, 932844), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 9, 15, 932844), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:09:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:09:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:14:15 UTC)" executed successfully builder stdout | 2023-01-17 05:09:17,662 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:17,671 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:17,671 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:17,677 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:28.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 05:09:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:28 UTC)" (scheduled at 2023-01-17 05:09:18.735066+00:00) notificationworker stdout | 2023-01-17 05:09:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 18, 735988), True, datetime.datetime(2023, 1, 17, 5, 9, 18, 735988), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:09:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:09:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:25.334138+00:00 (in 5.091433 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:09:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:20 UTC)" (scheduled at 2023-01-17 05:09:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:09:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:09:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 20, 243144), True, datetime.datetime(2023, 1, 17, 5, 9, 20, 243144), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:09:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:09:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:09:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:09:22,625 [244] [DEBUG] [app] Starting request: urn:request:190a2fa1-d435-49eb-91cb-d553e3c6bdf8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:22,625 [241] [DEBUG] [app] Starting request: urn:request:443b326b-1afc-45f7-b0e8-55288148af43 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:22,630 [232] [DEBUG] [app] Starting request: urn:request:f2d4dffc-8ac6-464b-ac0e-bc42e6c6a2a9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,630 [240] [DEBUG] [app] Starting request: urn:request:67963599-e9cd-4f99-abca-57b892628294 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,630 [232] [DEBUG] [app] Ending request: urn:request:f2d4dffc-8ac6-464b-ac0e-bc42e6c6a2a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2d4dffc-8ac6-464b-ac0e-bc42e6c6a2a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,631 [240] [DEBUG] [app] Ending request: urn:request:67963599-e9cd-4f99-abca-57b892628294 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:67963599-e9cd-4f99-abca-57b892628294', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:09:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:22,635 [239] [DEBUG] [app] Starting request: urn:request:7eb53272-3cfe-4097-8518-063755494ecf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:22,636 [239] [DEBUG] [app] Ending request: urn:request:7eb53272-3cfe-4097-8518-063755494ecf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7eb53272-3cfe-4097-8518-063755494ecf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:09:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:22,636 [244] [DEBUG] [app] Starting request: urn:request:81b18617-d5ae-4fba-9c2f-691c13d7ae84 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:22,636 [244] [DEBUG] [app] Ending request: urn:request:81b18617-d5ae-4fba-9c2f-691c13d7ae84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:81b18617-d5ae-4fba-9c2f-691c13d7ae84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:22,648 [244] [DEBUG] [app] Ending request: urn:request:190a2fa1-d435-49eb-91cb-d553e3c6bdf8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:190a2fa1-d435-49eb-91cb-d553e3c6bdf8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:22,648 [241] [DEBUG] [app] Ending request: urn:request:443b326b-1afc-45f7-b0e8-55288148af43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:443b326b-1afc-45f7-b0e8-55288148af43', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:22,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:09:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:09:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:39.230038+00:00 (in 16.392188 seconds) securityworker stdout | 2023-01-17 05:09:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:52 UTC)" (scheduled at 2023-01-17 05:09:22.837365+00:00) securityworker stdout | 2023-01-17 05:09:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:09:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:09:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:48.830274+00:00 (in 24.188042 seconds) exportactionlogsworker stdout | 2023-01-17 05:09:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:24 UTC)" (scheduled at 2023-01-17 05:09:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:09:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:09:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:09:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:09:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:20.242198+00:00 (in 54.907532 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:09:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:25 UTC)" (scheduled at 2023-01-17 05:09:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:09:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:09:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:09:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:09:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:57.226424+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 05:09:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:57 UTC)" (scheduled at 2023-01-17 05:09:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:09:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 9, 27, 227260), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:09:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:09:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:09:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:57 UTC)" executed successfully builder stdout | 2023-01-17 05:09:27,678 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:27,687 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:27,687 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:27,693 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:38.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:09:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:38 UTC)" (scheduled at 2023-01-17 05:09:28.735066+00:00) notificationworker stdout | 2023-01-17 05:09:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 28, 735979), True, datetime.datetime(2023, 1, 17, 5, 9, 28, 735979), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:09:29,312 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:09:29,533 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:09:29,823 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:09:30,585 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:09:30,603 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:09:31,823 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:09:32,650 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:09:32,675 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:09:33,075 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:09:33,872 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:09:34,037 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:09:34,999 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:09:35,602 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:09:35,607 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:09:35,609 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:09:35,624 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:09:35,792 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:09:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:09:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:05.848675+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:09:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:05 UTC)" (scheduled at 2023-01-17 05:09:35.848675+00:00) gcworker stdout | 2023-01-17 05:09:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722575849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:09:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:09:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:09:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:09:36,940 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:09:37,526 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:09:37,626 [244] [DEBUG] [app] Starting request: urn:request:2fb474e5-3a88-4dc1-9bb0-7e0d25db07d3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:37,626 [239] [DEBUG] [app] Starting request: urn:request:6b2a205f-8d3f-4437-ba12-3c2c0e27af66 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:37,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:37,631 [232] [DEBUG] [app] Starting request: urn:request:0e8ff44c-1760-49f1-afa9-5f5f70fd4ea2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:37,631 [242] [DEBUG] [app] Starting request: urn:request:7ad2df42-f2ac-4466-81f4-b03056b11f8a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:37,632 [232] [DEBUG] [app] Ending request: urn:request:0e8ff44c-1760-49f1-afa9-5f5f70fd4ea2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e8ff44c-1760-49f1-afa9-5f5f70fd4ea2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:37,632 [242] [DEBUG] [app] Ending request: urn:request:7ad2df42-f2ac-4466-81f4-b03056b11f8a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ad2df42-f2ac-4466-81f4-b03056b11f8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:09:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:09:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:37,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:37,636 [244] [DEBUG] [app] Starting request: urn:request:55d06eaa-a2e6-4f48-8ff8-033454bec10a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:37,637 [244] [DEBUG] [app] Ending request: urn:request:55d06eaa-a2e6-4f48-8ff8-033454bec10a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55d06eaa-a2e6-4f48-8ff8-033454bec10a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:37,637 [243] [DEBUG] [app] Starting request: urn:request:fd21c1d2-ec05-44ec-9416-96188126c7fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:37,638 [243] [DEBUG] [app] Ending request: urn:request:fd21c1d2-ec05-44ec-9416-96188126c7fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd21c1d2-ec05-44ec-9416-96188126c7fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:37,648 [244] [DEBUG] [app] Ending request: urn:request:2fb474e5-3a88-4dc1-9bb0-7e0d25db07d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2fb474e5-3a88-4dc1-9bb0-7e0d25db07d3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:37,648 [239] [DEBUG] [app] Ending request: urn:request:6b2a205f-8d3f-4437-ba12-3c2c0e27af66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b2a205f-8d3f-4437-ba12-3c2c0e27af66', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:09:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:09:37,700 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:37,708 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:37,708 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:37,714 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:09:37,918 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:09:37,921 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:09:37,924 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:09:37,927 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:09:37,929 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:09:37,931 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:09:37,934 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:09:37,936 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:09:37,941 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:09:38,196 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:09:38,203 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:09:38,205 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:09:38,207 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:09:38,210 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:09:38,284 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:09:38,344 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:09:38,371 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:09:38,415 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:09:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:43.935629+00:00 (in 5.200091 seconds) notificationworker stdout | 2023-01-17 05:09:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:48 UTC)" (scheduled at 2023-01-17 05:09:38.735066+00:00) notificationworker stdout | 2023-01-17 05:09:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 38, 735940), True, datetime.datetime(2023, 1, 17, 5, 9, 38, 735940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:09:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:09:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:48.738458+00:00 (in 9.707203 seconds) repositorygcworker stdout | 2023-01-17 05:09:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:39 UTC)" (scheduled at 2023-01-17 05:09:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:09:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:09:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 39, 31692), True, datetime.datetime(2023, 1, 17, 5, 9, 39, 31692), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:09:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:09:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:09:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:09:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:52.837365+00:00 (in 13.606819 seconds) securityworker stdout | 2023-01-17 05:09:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:09 UTC)" (scheduled at 2023-01-17 05:09:39.230038+00:00) securityworker stdout | 2023-01-17 05:09:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:09:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:09:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:09:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:09:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:48.735066+00:00 (in 4.798933 seconds) notificationworker stdout | 2023-01-17 05:09:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:43 UTC)" (scheduled at 2023-01-17 05:09:43.935629+00:00) notificationworker stdout | 2023-01-17 05:09:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:09:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:43 UTC)" executed successfully builder stdout | 2023-01-17 05:09:47,724 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:47,733 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:47,733 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:47,738 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:09:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:09:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:59.337413+00:00 (in 11.404441 seconds) namespacegcworker stdout | 2023-01-17 05:09:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:47 UTC)" (scheduled at 2023-01-17 05:09:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:09:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:09:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:09:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:58.735066+00:00 (in 9.999424 seconds) notificationworker stdout | 2023-01-17 05:09:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:58 UTC)" (scheduled at 2023-01-17 05:09:48.735066+00:00) notificationworker stdout | 2023-01-17 05:09:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 48, 736067), True, datetime.datetime(2023, 1, 17, 5, 9, 48, 736067), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:09:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:09:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:39.030814+00:00 (in 50.291887 seconds) repositorygcworker stdout | 2023-01-17 05:09:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:48 UTC)" (scheduled at 2023-01-17 05:09:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:09:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:09:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:09:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:09:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:09:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:24.641816+00:00 (in 35.811084 seconds) exportactionlogsworker stdout | 2023-01-17 05:09:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:48 UTC)" (scheduled at 2023-01-17 05:09:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:09:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:09:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 48, 831162), True, datetime.datetime(2023, 1, 17, 5, 9, 48, 831162), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:09:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:09:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:09:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:09:52,625 [244] [DEBUG] [app] Starting request: urn:request:76966216-acf6-4883-8070-2febc53ac972 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:52,625 [243] [DEBUG] [app] Starting request: urn:request:71ed6a55-7a13-4a4e-818f-c0fde0d4f486 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:09:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:52,630 [232] [DEBUG] [app] Starting request: urn:request:f6584f33-9d3a-44b7-842f-cde9b9615446 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,630 [242] [DEBUG] [app] Starting request: urn:request:b70be41f-5c29-4e9c-bf12-62708b124c40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,631 [232] [DEBUG] [app] Ending request: urn:request:f6584f33-9d3a-44b7-842f-cde9b9615446 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f6584f33-9d3a-44b7-842f-cde9b9615446', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,631 [242] [DEBUG] [app] Ending request: urn:request:b70be41f-5c29-4e9c-bf12-62708b124c40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b70be41f-5c29-4e9c-bf12-62708b124c40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:09:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:09:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:52,636 [239] [DEBUG] [app] Starting request: urn:request:1c4a5790-1080-4c7a-9754-75bb144644b7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:52,636 [244] [DEBUG] [app] Starting request: urn:request:21b113e2-6c4f-4f38-b025-72a62acb3628 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:52,636 [239] [DEBUG] [app] Ending request: urn:request:1c4a5790-1080-4c7a-9754-75bb144644b7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1c4a5790-1080-4c7a-9754-75bb144644b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:52,637 [244] [DEBUG] [app] Ending request: urn:request:21b113e2-6c4f-4f38-b025-72a62acb3628 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21b113e2-6c4f-4f38-b025-72a62acb3628', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:52,648 [243] [DEBUG] [app] Ending request: urn:request:71ed6a55-7a13-4a4e-818f-c0fde0d4f486 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71ed6a55-7a13-4a4e-818f-c0fde0d4f486', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:52,648 [244] [DEBUG] [app] Ending request: urn:request:76966216-acf6-4883-8070-2febc53ac972 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76966216-acf6-4883-8070-2febc53ac972', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:09:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:09:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:09.230038+00:00 (in 16.392215 seconds) securityworker stdout | 2023-01-17 05:09:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:22 UTC)" (scheduled at 2023-01-17 05:09:52.837365+00:00) securityworker stdout | 2023-01-17 05:09:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:09:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:09:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:27.226424+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:09:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:27 UTC)" (scheduled at 2023-01-17 05:09:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:09:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 9, 57, 227265), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:09:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:09:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:09:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:27 UTC)" executed successfully builder stdout | 2023-01-17 05:09:57,744 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:57,752 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:57,752 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:57,758 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:08.735066+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:09:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:08 UTC)" (scheduled at 2023-01-17 05:09:58.735066+00:00) notificationworker stdout | 2023-01-17 05:09:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 58, 735949), True, datetime.datetime(2023, 1, 17, 5, 9, 58, 735949), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:58,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:58,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:09:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:09:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:47.932490+00:00 (in 48.594622 seconds) namespacegcworker stdout | 2023-01-17 05:09:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:59 UTC)" (scheduled at 2023-01-17 05:09:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:09:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:09:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 59, 338258), True, datetime.datetime(2023, 1, 17, 5, 9, 59, 338258), 0, 'namespacegc/%', 50, 1, 0]) securityworker stdout | 2023-01-17 05:09:59,344 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} namespacegcworker stdout | 2023-01-17 05:09:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:09:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:09:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:59 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:09:59,539 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:09:59,855 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:10:00,621 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:10:00,628 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:10:01,844 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:10:02,677 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:10:02,702 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:10:03,110 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:10:03,908 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:10:04,072 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:10:05,035 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:10:05,608 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:10:05,613 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:10:05,615 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:10:05,659 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:10:05,808 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:10:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:10:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:35.848675+00:00 (in 29.999513 seconds) gcworker stdout | 2023-01-17 05:10:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:35 UTC)" (scheduled at 2023-01-17 05:10:05.848675+00:00) gcworker stdout | 2023-01-17 05:10:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722605849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:10:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:10:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:10:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:10:06,962 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:10:07,561 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:10:07,625 [244] [DEBUG] [app] Starting request: urn:request:1a9298b1-5b8d-46a0-ab43-ca3fd8a292fd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:07,626 [239] [DEBUG] [app] Starting request: urn:request:3423f139-ff0c-4dae-9401-44ee9fb0ef98 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:07,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:07,631 [232] [DEBUG] [app] Starting request: urn:request:4f50390f-d939-4198-8b15-3b9c5dba0014 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,632 [240] [DEBUG] [app] Starting request: urn:request:4ffe37b5-36c8-4d5d-88a9-68a8a20d4965 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,632 [232] [DEBUG] [app] Ending request: urn:request:4f50390f-d939-4198-8b15-3b9c5dba0014 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f50390f-d939-4198-8b15-3b9c5dba0014', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:10:07,632 [240] [DEBUG] [app] Ending request: urn:request:4ffe37b5-36c8-4d5d-88a9-68a8a20d4965 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ffe37b5-36c8-4d5d-88a9-68a8a20d4965', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:10:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:07,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:07,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:07,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:07,637 [243] [DEBUG] [app] Starting request: urn:request:676c49d3-7824-4241-9a46-bd0801397a2f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:07,637 [243] [DEBUG] [app] Ending request: urn:request:676c49d3-7824-4241-9a46-bd0801397a2f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:676c49d3-7824-4241-9a46-bd0801397a2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:07,638 [241] [DEBUG] [app] Starting request: urn:request:ba79cd58-9832-4f54-8bc0-b85413058c66 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:07,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:07,638 [241] [DEBUG] [app] Ending request: urn:request:ba79cd58-9832-4f54-8bc0-b85413058c66 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba79cd58-9832-4f54-8bc0-b85413058c66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:07,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:07,639 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:10:07,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:07,649 [244] [DEBUG] [app] Ending request: urn:request:1a9298b1-5b8d-46a0-ab43-ca3fd8a292fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1a9298b1-5b8d-46a0-ab43-ca3fd8a292fd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:07,649 [239] [DEBUG] [app] Ending request: urn:request:3423f139-ff0c-4dae-9401-44ee9fb0ef98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3423f139-ff0c-4dae-9401-44ee9fb0ef98', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:07,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:07,649 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.025) gunicorn-web stdout | 2023-01-17 05:10:07,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:10:07,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:10:07,768 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:07,776 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:07,776 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:07,782 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:10:07,926 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:10:07,929 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:10:07,932 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:10:07,935 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:10:07,937 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:10:07,940 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:10:07,942 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:10:07,944 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:10:07,951 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:10:08,202 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:10:08,208 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:10:08,211 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:10:08,214 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:10:08,216 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:10:08,289 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:10:08,379 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:10:08,404 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:10:08,450 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:10:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:18.735066+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:10:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:18 UTC)" (scheduled at 2023-01-17 05:10:08.735066+00:00) notificationworker stdout | 2023-01-17 05:10:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 8, 735930), True, datetime.datetime(2023, 1, 17, 5, 10, 8, 735930), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:22.837365+00:00 (in 13.606892 seconds) securityworker stdout | 2023-01-17 05:10:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:39 UTC)" (scheduled at 2023-01-17 05:10:09.230038+00:00) securityworker stdout | 2023-01-17 05:10:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:10:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:10:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:10:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:10:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:10:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:10:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:39 UTC)" executed successfully builder stdout | 2023-01-17 05:10:17,784 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:17,793 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:17,793 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:17,799 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:28.735066+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:10:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:28 UTC)" (scheduled at 2023-01-17 05:10:18.735066+00:00) notificationworker stdout | 2023-01-17 05:10:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 18, 735980), True, datetime.datetime(2023, 1, 17, 5, 10, 18, 735980), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:10:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:10:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:25.334138+00:00 (in 5.091430 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:10:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:20 UTC)" (scheduled at 2023-01-17 05:10:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:10:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:10:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 20, 243087), True, datetime.datetime(2023, 1, 17, 5, 10, 20, 243087), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:10:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:10:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:10:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:10:22,626 [241] [DEBUG] [app] Starting request: urn:request:d1745c40-4e45-42ca-82a0-621d77c9f735 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:22,626 [243] [DEBUG] [app] Starting request: urn:request:21e1d8f9-1cd0-49a8-824c-a7477490c049 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:22,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:22,631 [232] [DEBUG] [app] Starting request: urn:request:fe909f1e-19d7-4819-99f8-9d9154e21a6a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:22,632 [236] [DEBUG] [app] Starting request: urn:request:26205826-b02d-479f-862d-5474b186ab8c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:22,632 [232] [DEBUG] [app] Ending request: urn:request:fe909f1e-19d7-4819-99f8-9d9154e21a6a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fe909f1e-19d7-4819-99f8-9d9154e21a6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:10:22,632 [236] [DEBUG] [app] Ending request: urn:request:26205826-b02d-479f-862d-5474b186ab8c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26205826-b02d-479f-862d-5474b186ab8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:10:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:10:22,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:10:22,633 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,635 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:22,637 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:22,637 [243] [DEBUG] [app] Starting request: urn:request:740b8e38-652a-474c-9f1b-d938df9ea76a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:22,637 [243] [DEBUG] [app] Ending request: urn:request:740b8e38-652a-474c-9f1b-d938df9ea76a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:740b8e38-652a-474c-9f1b-d938df9ea76a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:22,638 [239] [DEBUG] [app] Starting request: urn:request:55faea00-3707-411f-893d-86e4fe22b7ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:22,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:22,638 [239] [DEBUG] [app] Ending request: urn:request:55faea00-3707-411f-893d-86e4fe22b7ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55faea00-3707-411f-893d-86e4fe22b7ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:22,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:22,639 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:22,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:22,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:22,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:22,649 [243] [DEBUG] [app] Ending request: urn:request:21e1d8f9-1cd0-49a8-824c-a7477490c049 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21e1d8f9-1cd0-49a8-824c-a7477490c049', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:22,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:10:22,649 [241] [DEBUG] [app] Ending request: urn:request:d1745c40-4e45-42ca-82a0-621d77c9f735 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1745c40-4e45-42ca-82a0-621d77c9f735', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:22,649 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 05:10:22,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:10:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:39.230038+00:00 (in 16.392204 seconds) securityworker stdout | 2023-01-17 05:10:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:52 UTC)" (scheduled at 2023-01-17 05:10:22.837365+00:00) securityworker stdout | 2023-01-17 05:10:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:10:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:10:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:48.830274+00:00 (in 24.188007 seconds) exportactionlogsworker stdout | 2023-01-17 05:10:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:24 UTC)" (scheduled at 2023-01-17 05:10:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:10:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:10:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:10:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:10:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:20.242198+00:00 (in 54.907578 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:10:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:25 UTC)" (scheduled at 2023-01-17 05:10:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:10:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:10:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:10:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:10:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:57.226424+00:00 (in 29.999554 seconds) buildlogsarchiver stdout | 2023-01-17 05:10:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:57 UTC)" (scheduled at 2023-01-17 05:10:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:10:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 10, 27, 227237), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:10:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:10:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:10:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:57 UTC)" executed successfully builder stdout | 2023-01-17 05:10:27,809 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:27,818 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:27,818 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:27,824 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:38.735066+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:10:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:38 UTC)" (scheduled at 2023-01-17 05:10:28.735066+00:00) notificationworker stdout | 2023-01-17 05:10:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 28, 735953), True, datetime.datetime(2023, 1, 17, 5, 10, 28, 735953), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:29,357 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:10:29,574 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:10:29,867 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:10:30,638 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:10:30,641 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:10:31,859 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:10:32,687 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:10:32,716 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:10:33,132 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:10:33,934 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:10:34,079 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:10:35,071 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:10:35,615 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:10:35,619 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:10:35,621 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:10:35,694 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:10:35,843 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gcworker stdout | 2023-01-17 05:10:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:10:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:05.848675+00:00 (in 29.999571 seconds) gcworker stdout | 2023-01-17 05:10:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:05 UTC)" (scheduled at 2023-01-17 05:10:35.848675+00:00) gcworker stdout | 2023-01-17 05:10:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722635849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:10:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:10:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:10:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:05 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:10:36,983 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:10:37,590 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:10:37,625 [243] [DEBUG] [app] Starting request: urn:request:8a7ea4dc-2714-4498-a671-357464779beb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:37,626 [244] [DEBUG] [app] Starting request: urn:request:3564788d-1269-4d3b-a117-d1b73f009148 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:37,630 [240] [DEBUG] [app] Starting request: urn:request:d0491979-7c3a-4492-a8b8-bed7ea978a37 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:37,630 [232] [DEBUG] [app] Starting request: urn:request:45f86b0e-38ec-4084-b9b7-9c8f4017c653 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:37,631 [240] [DEBUG] [app] Ending request: urn:request:d0491979-7c3a-4492-a8b8-bed7ea978a37 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d0491979-7c3a-4492-a8b8-bed7ea978a37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:37,631 [232] [DEBUG] [app] Ending request: urn:request:45f86b0e-38ec-4084-b9b7-9c8f4017c653 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45f86b0e-38ec-4084-b9b7-9c8f4017c653', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:10:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:10:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:37,636 [239] [DEBUG] [app] Starting request: urn:request:a3e7befc-f4ff-4060-a7ce-e81aa2e221db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:37,636 [241] [DEBUG] [app] Starting request: urn:request:2e1de5eb-ef20-47e9-a25a-d55ee56b3a74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:37,637 [239] [DEBUG] [app] Ending request: urn:request:a3e7befc-f4ff-4060-a7ce-e81aa2e221db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3e7befc-f4ff-4060-a7ce-e81aa2e221db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:37,637 [241] [DEBUG] [app] Ending request: urn:request:2e1de5eb-ef20-47e9-a25a-d55ee56b3a74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e1de5eb-ef20-47e9-a25a-d55ee56b3a74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:10:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:37,648 [243] [DEBUG] [app] Ending request: urn:request:8a7ea4dc-2714-4498-a671-357464779beb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a7ea4dc-2714-4498-a671-357464779beb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:37,648 [244] [DEBUG] [app] Ending request: urn:request:3564788d-1269-4d3b-a117-d1b73f009148 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3564788d-1269-4d3b-a117-d1b73f009148', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:37,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:10:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:10:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:10:37,834 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:37,842 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:37,842 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:37,848 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:10:37,934 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:10:37,937 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:10:37,940 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:10:37,943 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:10:37,945 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:10:37,948 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:10:37,950 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:10:37,953 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:10:37,959 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:10:38,208 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:10:38,214 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:10:38,218 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:10:38,221 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:10:38,223 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:10:38,324 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:10:38,402 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:10:38,420 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:10:38,480 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:10:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:43.935629+00:00 (in 5.200119 seconds) notificationworker stdout | 2023-01-17 05:10:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:48 UTC)" (scheduled at 2023-01-17 05:10:38.735066+00:00) notificationworker stdout | 2023-01-17 05:10:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 38, 735922), True, datetime.datetime(2023, 1, 17, 5, 10, 38, 735922), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:10:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:10:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:48.738458+00:00 (in 9.707209 seconds) repositorygcworker stdout | 2023-01-17 05:10:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:39 UTC)" (scheduled at 2023-01-17 05:10:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:10:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:10:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 39, 31691), True, datetime.datetime(2023, 1, 17, 5, 10, 39, 31691), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:10:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:10:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:10:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:52.837365+00:00 (in 13.606865 seconds) securityworker stdout | 2023-01-17 05:10:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:09 UTC)" (scheduled at 2023-01-17 05:10:39.230038+00:00) securityworker stdout | 2023-01-17 05:10:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:10:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:10:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:10:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:10:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:10:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:10:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:10:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:48.735066+00:00 (in 4.798989 seconds) notificationworker stdout | 2023-01-17 05:10:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:43 UTC)" (scheduled at 2023-01-17 05:10:43.935629+00:00) notificationworker stdout | 2023-01-17 05:10:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:10:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:43 UTC)" executed successfully builder stdout | 2023-01-17 05:10:47,854 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:47,863 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:47,863 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:47,868 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:10:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:10:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:59.337413+00:00 (in 11.404481 seconds) namespacegcworker stdout | 2023-01-17 05:10:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:47 UTC)" (scheduled at 2023-01-17 05:10:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:10:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:10:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:10:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:58.735066+00:00 (in 9.999479 seconds) notificationworker stdout | 2023-01-17 05:10:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:58 UTC)" (scheduled at 2023-01-17 05:10:48.735066+00:00) notificationworker stdout | 2023-01-17 05:10:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 48, 736002), True, datetime.datetime(2023, 1, 17, 5, 10, 48, 736002), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:10:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:10:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:39.030814+00:00 (in 50.291884 seconds) repositorygcworker stdout | 2023-01-17 05:10:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:48 UTC)" (scheduled at 2023-01-17 05:10:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:10:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:10:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:10:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:10:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:10:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:24.641816+00:00 (in 35.811079 seconds) exportactionlogsworker stdout | 2023-01-17 05:10:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:48 UTC)" (scheduled at 2023-01-17 05:10:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:10:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:10:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 48, 831121), True, datetime.datetime(2023, 1, 17, 5, 10, 48, 831121), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:10:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:10:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:10:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:48 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:10:50,626 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 05:10:50,627 [72] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2023-01-17 05:10:50 UTC)" (scheduled at 2023-01-17 05:10:50.626417+00:00) manifestbackfillworker stdout | 2023-01-17 05:10:50,627 [72] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0]) manifestbackfillworker stdout | 2023-01-17 05:10:50,628 [72] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:50.626417+00:00 (in 3599.999599 seconds) manifestbackfillworker stdout | 2023-01-17 05:10:50,638 [72] [DEBUG] [__main__] Manifest backfill worker has completed; skipping manifestbackfillworker stdout | 2023-01-17 05:10:50,638 [72] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 05:10:50,638 [72] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2023-01-17 06:10:50 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:10:52,626 [243] [DEBUG] [app] Starting request: urn:request:227f110e-5043-494e-b683-adcca2762ba1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:52,626 [241] [DEBUG] [app] Starting request: urn:request:0b3670d2-52a3-4c9f-ab2b-6a76e12d4314 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:10:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:52,630 [232] [DEBUG] [app] Starting request: urn:request:0a6f4a43-2a8f-496b-97a3-a2a9e6688c9e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,631 [232] [DEBUG] [app] Ending request: urn:request:0a6f4a43-2a8f-496b-97a3-a2a9e6688c9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a6f4a43-2a8f-496b-97a3-a2a9e6688c9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,631 [242] [DEBUG] [app] Starting request: urn:request:312478be-119f-42e1-9db8-ddd4144aabc0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:10:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:10:52,631 [242] [DEBUG] [app] Ending request: urn:request:312478be-119f-42e1-9db8-ddd4144aabc0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:312478be-119f-42e1-9db8-ddd4144aabc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:10:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:52,635 [239] [DEBUG] [app] Starting request: urn:request:56c6f4d9-9383-4c58-b69d-ccbcbd3f3d5a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:52,636 [239] [DEBUG] [app] Ending request: urn:request:56c6f4d9-9383-4c58-b69d-ccbcbd3f3d5a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56c6f4d9-9383-4c58-b69d-ccbcbd3f3d5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:10:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:52,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:52,637 [239] [DEBUG] [app] Starting request: urn:request:70733e80-f5cc-4e76-9d28-2b30bfdf27a3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:52,637 [239] [DEBUG] [app] Ending request: urn:request:70733e80-f5cc-4e76-9d28-2b30bfdf27a3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70733e80-f5cc-4e76-9d28-2b30bfdf27a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:52,647 [243] [DEBUG] [app] Ending request: urn:request:227f110e-5043-494e-b683-adcca2762ba1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:227f110e-5043-494e-b683-adcca2762ba1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:10:52,648 [241] [DEBUG] [app] Ending request: urn:request:0b3670d2-52a3-4c9f-ab2b-6a76e12d4314 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b3670d2-52a3-4c9f-ab2b-6a76e12d4314', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:10:52,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:10:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:10:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:09.230038+00:00 (in 16.392188 seconds) securityworker stdout | 2023-01-17 05:10:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:22 UTC)" (scheduled at 2023-01-17 05:10:52.837365+00:00) securityworker stdout | 2023-01-17 05:10:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:10:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:10:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:27.226424+00:00 (in 29.999507 seconds) buildlogsarchiver stdout | 2023-01-17 05:10:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:27 UTC)" (scheduled at 2023-01-17 05:10:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:10:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 10, 57, 227271), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:10:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:10:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:10:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:27 UTC)" executed successfully builder stdout | 2023-01-17 05:10:57,879 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:57,888 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:57,888 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:57,894 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:08.735066+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:10:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:08 UTC)" (scheduled at 2023-01-17 05:10:58.735066+00:00) notificationworker stdout | 2023-01-17 05:10:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 58, 735939), True, datetime.datetime(2023, 1, 17, 5, 10, 58, 735939), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:10:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:10:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:47.932490+00:00 (in 48.594597 seconds) namespacegcworker stdout | 2023-01-17 05:10:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:59 UTC)" (scheduled at 2023-01-17 05:10:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:10:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:10:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 59, 338333), True, datetime.datetime(2023, 1, 17, 5, 10, 59, 338333), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:10:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:10:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:10:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:59,392 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:10:59,605 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:10:59,892 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:11:00,674 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:11:00,676 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:11:01,865 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:11:02,717 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:11:02,721 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:11:03,140 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:11:03,970 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:11:04,092 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:11:05,084 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:11:05,621 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:11:05,625 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:11:05,627 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:11:05,730 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gcworker stdout | 2023-01-17 05:11:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:11:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:35.848675+00:00 (in 29.999396 seconds) gcworker stdout | 2023-01-17 05:11:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:35 UTC)" (scheduled at 2023-01-17 05:11:05.848675+00:00) gcworker stdout | 2023-01-17 05:11:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722665849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:11:05,865 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:11:05,866 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:11:05,866 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:35 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:11:05,878 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:11:07,000 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:11:07,605 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:11:07,625 [244] [DEBUG] [app] Starting request: urn:request:d218dfc7-e526-467b-b85f-7d8043dc1b76 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:07,625 [239] [DEBUG] [app] Starting request: urn:request:f521f133-cfdb-4915-95cf-d04cf625f481 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:07,630 [242] [DEBUG] [app] Starting request: urn:request:1f86ee6a-99dc-4da4-9853-cb3ff8dd4a20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:07,631 [237] [DEBUG] [app] Starting request: urn:request:c7552d7e-1bc4-4d9f-aadd-d0eaf15c5ee5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:07,631 [242] [DEBUG] [app] Ending request: urn:request:1f86ee6a-99dc-4da4-9853-cb3ff8dd4a20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1f86ee6a-99dc-4da4-9853-cb3ff8dd4a20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:11:07,631 [237] [DEBUG] [app] Ending request: urn:request:c7552d7e-1bc4-4d9f-aadd-d0eaf15c5ee5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c7552d7e-1bc4-4d9f-aadd-d0eaf15c5ee5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:11:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:07,636 [239] [DEBUG] [app] Starting request: urn:request:c5e5887d-d1e2-4c37-a9f9-9190fc7e1c9b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:07,636 [239] [DEBUG] [app] Ending request: urn:request:c5e5887d-d1e2-4c37-a9f9-9190fc7e1c9b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5e5887d-d1e2-4c37-a9f9-9190fc7e1c9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:07,637 [243] [DEBUG] [app] Starting request: urn:request:82cc3f87-6290-4ca1-afd0-207f2e31d4f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:07,637 [243] [DEBUG] [app] Ending request: urn:request:82cc3f87-6290-4ca1-afd0-207f2e31d4f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82cc3f87-6290-4ca1-afd0-207f2e31d4f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:07,648 [239] [DEBUG] [app] Ending request: urn:request:f521f133-cfdb-4915-95cf-d04cf625f481 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f521f133-cfdb-4915-95cf-d04cf625f481', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:11:07,649 [244] [DEBUG] [app] Ending request: urn:request:d218dfc7-e526-467b-b85f-7d8043dc1b76 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d218dfc7-e526-467b-b85f-7d8043dc1b76', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:07,649 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 05:11:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:11:07,904 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:07,913 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:07,913 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:07,919 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:11:07,942 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:11:07,945 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:11:07,947 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:11:07,950 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:11:07,953 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:11:07,955 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:11:07,958 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:11:07,960 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:11:07,966 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:11:08,214 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:11:08,220 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:11:08,224 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:11:08,227 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:11:08,230 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:11:08,345 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:11:08,412 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:11:08,455 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:11:08,516 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:11:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:18.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 05:11:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:18 UTC)" (scheduled at 2023-01-17 05:11:08.735066+00:00) notificationworker stdout | 2023-01-17 05:11:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 8, 735925), True, datetime.datetime(2023, 1, 17, 5, 11, 8, 735925), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:22.837365+00:00 (in 13.606867 seconds) securityworker stdout | 2023-01-17 05:11:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:39 UTC)" (scheduled at 2023-01-17 05:11:09.230038+00:00) securityworker stdout | 2023-01-17 05:11:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:11:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:11:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:09,235 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:09,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:09,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:11:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:39 UTC)" executed successfully builder stdout | 2023-01-17 05:11:17,929 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:17,941 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:17,941 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:17,946 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:28.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:11:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:28 UTC)" (scheduled at 2023-01-17 05:11:18.735066+00:00) notificationworker stdout | 2023-01-17 05:11:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 18, 735933), True, datetime.datetime(2023, 1, 17, 5, 11, 18, 735933), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:11:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:11:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:25.334138+00:00 (in 5.091434 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:11:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:20 UTC)" (scheduled at 2023-01-17 05:11:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:11:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:11:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 20, 243120), True, datetime.datetime(2023, 1, 17, 5, 11, 20, 243120), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:11:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:11:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:11:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:11:22,625 [239] [DEBUG] [app] Starting request: urn:request:6b843671-c556-4c3f-bc8a-9a44be13e900 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:22,625 [244] [DEBUG] [app] Starting request: urn:request:70a4a064-f5e5-4015-a894-4fbd46bff794 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:22,630 [242] [DEBUG] [app] Starting request: urn:request:dec54383-f139-43d5-bc24-223a4c9f82b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,631 [232] [DEBUG] [app] Starting request: urn:request:d52d5605-9bb7-40ae-b11b-8f30292f9361 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,631 [242] [DEBUG] [app] Ending request: urn:request:dec54383-f139-43d5-bc24-223a4c9f82b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dec54383-f139-43d5-bc24-223a4c9f82b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:11:22,631 [232] [DEBUG] [app] Ending request: urn:request:d52d5605-9bb7-40ae-b11b-8f30292f9361 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d52d5605-9bb7-40ae-b11b-8f30292f9361', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:11:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:11:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:22,635 [243] [DEBUG] [app] Starting request: urn:request:cee5f96f-5b10-4542-b188-a676c7833881 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:22,636 [244] [DEBUG] [app] Starting request: urn:request:a53a3ab7-bad7-4682-a0ef-1c24a9466188 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:22,636 [244] [DEBUG] [app] Ending request: urn:request:a53a3ab7-bad7-4682-a0ef-1c24a9466188 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a53a3ab7-bad7-4682-a0ef-1c24a9466188', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:22,636 [243] [DEBUG] [app] Ending request: urn:request:cee5f96f-5b10-4542-b188-a676c7833881 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cee5f96f-5b10-4542-b188-a676c7833881', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:11:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:22,647 [239] [DEBUG] [app] Ending request: urn:request:6b843671-c556-4c3f-bc8a-9a44be13e900 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b843671-c556-4c3f-bc8a-9a44be13e900', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:22,647 [244] [DEBUG] [app] Ending request: urn:request:70a4a064-f5e5-4015-a894-4fbd46bff794 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:70a4a064-f5e5-4015-a894-4fbd46bff794', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:11:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:11:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:39.230038+00:00 (in 16.392171 seconds) securityworker stdout | 2023-01-17 05:11:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:52 UTC)" (scheduled at 2023-01-17 05:11:22.837365+00:00) securityworker stdout | 2023-01-17 05:11:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:22,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:11:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:11:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:44.641497+00:00 (in 19.999214 seconds) exportactionlogsworker stdout | 2023-01-17 05:11:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:24 UTC)" (scheduled at 2023-01-17 05:11:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:11:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:11:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:11:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:11:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:39.333860+00:00 (in 13.999235 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:11:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:25 UTC)" (scheduled at 2023-01-17 05:11:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:11:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:11:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:11:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:11:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:57.226424+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 05:11:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:57 UTC)" (scheduled at 2023-01-17 05:11:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:11:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 11, 27, 227300), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:11:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:11:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:11:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:57 UTC)" executed successfully builder stdout | 2023-01-17 05:11:27,947 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:27,955 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:27,955 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:27,961 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:38.735066+00:00 (in 9.999479 seconds) notificationworker stdout | 2023-01-17 05:11:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:38 UTC)" (scheduled at 2023-01-17 05:11:28.735066+00:00) notificationworker stdout | 2023-01-17 05:11:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 28, 736023), True, datetime.datetime(2023, 1, 17, 5, 11, 28, 736023), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:29,410 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:11:29,633 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:11:29,927 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:11:30,682 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:11:30,708 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:11:31,900 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:11:32,754 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:11:32,756 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:11:33,175 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:11:34,005 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:11:34,127 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:11:35,101 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:11:35,628 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:11:35,632 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:11:35,634 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:11:35,766 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gcworker stdout | 2023-01-17 05:11:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:11:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:05.848675+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:11:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:05 UTC)" (scheduled at 2023-01-17 05:11:35.848675+00:00) gcworker stdout | 2023-01-17 05:11:35,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:11:35,856 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722695856, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:11:35,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:11:35,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:11:35,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:05 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:11:35,893 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:11:37,036 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:11:37,625 [241] [DEBUG] [app] Starting request: urn:request:2e37c2ae-716f-4322-a192-f16191307e14 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:37,625 [244] [DEBUG] [app] Starting request: urn:request:c77608ae-2292-4d75-a008-cfa646d0d30e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:37,630 [240] [DEBUG] [app] Starting request: urn:request:4d60e0ab-e5a0-43a2-a9d5-f34ebdea669b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,631 [235] [DEBUG] [app] Starting request: urn:request:bdbff719-451d-42da-8db5-ab5d7b778c35 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,631 [240] [DEBUG] [app] Ending request: urn:request:4d60e0ab-e5a0-43a2-a9d5-f34ebdea669b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d60e0ab-e5a0-43a2-a9d5-f34ebdea669b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:11:37,631 [235] [DEBUG] [app] Ending request: urn:request:bdbff719-451d-42da-8db5-ab5d7b778c35 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bdbff719-451d-42da-8db5-ab5d7b778c35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:11:37,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:11:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:37,635 [239] [DEBUG] [app] Starting request: urn:request:c365f038-9486-4b0d-9079-0c97126d09a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:37,635 [244] [DEBUG] [app] Starting request: urn:request:d5a9a8e2-ede1-4ea6-bbde-91fabd19dadf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:37,636 [239] [DEBUG] [app] Ending request: urn:request:c365f038-9486-4b0d-9079-0c97126d09a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c365f038-9486-4b0d-9079-0c97126d09a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:37,636 [244] [DEBUG] [app] Ending request: urn:request:d5a9a8e2-ede1-4ea6-bbde-91fabd19dadf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5a9a8e2-ede1-4ea6-bbde-91fabd19dadf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql notificationworker stdout | 2023-01-17 05:11:37,641 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:11:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:37,647 [244] [DEBUG] [app] Ending request: urn:request:c77608ae-2292-4d75-a008-cfa646d0d30e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c77608ae-2292-4d75-a008-cfa646d0d30e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:37,647 [241] [DEBUG] [app] Ending request: urn:request:2e37c2ae-716f-4322-a192-f16191307e14 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e37c2ae-716f-4322-a192-f16191307e14', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:11:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:11:37,950 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:11:37,953 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:11:37,956 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:11:37,959 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:11:37,961 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:11:37,963 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} builder stdout | 2023-01-17 05:11:37,966 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-registry stdout | 2023-01-17 05:11:37,966 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:11:37,970 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:11:37,974 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} builder stdout | 2023-01-17 05:11:37,974 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:37,975 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:37,980 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:38,220 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:11:38,224 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:11:38,231 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:11:38,233 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:11:38,235 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:11:38,381 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:11:38,444 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:11:38,482 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:11:38,539 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:11:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:43.935629+00:00 (in 5.200143 seconds) notificationworker stdout | 2023-01-17 05:11:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:48 UTC)" (scheduled at 2023-01-17 05:11:38.735066+00:00) notificationworker stdout | 2023-01-17 05:11:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 38, 735875), True, datetime.datetime(2023, 1, 17, 5, 11, 38, 735875), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:11:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:11:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:48.738458+00:00 (in 9.707214 seconds) repositorygcworker stdout | 2023-01-17 05:11:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:39 UTC)" (scheduled at 2023-01-17 05:11:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:11:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:11:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 39, 31672), True, datetime.datetime(2023, 1, 17, 5, 11, 39, 31672), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:11:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:11:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:11:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:52.837365+00:00 (in 13.606887 seconds) securityworker stdout | 2023-01-17 05:11:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:09 UTC)" (scheduled at 2023-01-17 05:11:39.230038+00:00) securityworker stdout | 2023-01-17 05:11:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:11:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:11:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:11:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:11:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:11:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:20.242198+00:00 (in 40.907776 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:11:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:39 UTC)" (scheduled at 2023-01-17 05:11:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:11:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 11, 39, 334779), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:11:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 39, 334779), True, datetime.datetime(2023, 1, 17, 5, 11, 39, 334779), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:11:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 39, 334779), True, datetime.datetime(2023, 1, 17, 5, 11, 39, 334779), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 11, 39, 334779), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:11:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:11:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:11:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:48.735066+00:00 (in 4.798955 seconds) notificationworker stdout | 2023-01-17 05:11:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:43 UTC)" (scheduled at 2023-01-17 05:11:43.935629+00:00) notificationworker stdout | 2023-01-17 05:11:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:11:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:11:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:11:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:48.830274+00:00 (in 4.188259 seconds) exportactionlogsworker stdout | 2023-01-17 05:11:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:44 UTC)" (scheduled at 2023-01-17 05:11:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:11:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 11, 44, 642441), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:11:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 44, 642441), True, datetime.datetime(2023, 1, 17, 5, 11, 44, 642441), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:11:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 44, 642441), True, datetime.datetime(2023, 1, 17, 5, 11, 44, 642441), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 11, 44, 642441), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:11:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:11:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:11:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:11:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:59.337413+00:00 (in 11.404467 seconds) namespacegcworker stdout | 2023-01-17 05:11:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:47 UTC)" (scheduled at 2023-01-17 05:11:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:11:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:11:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:47 UTC)" executed successfully builder stdout | 2023-01-17 05:11:47,990 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:47,999 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:47,999 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:48,005 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:58.735066+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:11:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:58 UTC)" (scheduled at 2023-01-17 05:11:48.735066+00:00) notificationworker stdout | 2023-01-17 05:11:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 48, 735979), True, datetime.datetime(2023, 1, 17, 5, 11, 48, 735979), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:11:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:11:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:04.738062+00:00 (in 15.999130 seconds) repositorygcworker stdout | 2023-01-17 05:11:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:48 UTC)" (scheduled at 2023-01-17 05:11:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:11:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:11:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:11:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:11:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:11:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:24.641816+00:00 (in 35.811018 seconds) exportactionlogsworker stdout | 2023-01-17 05:11:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:48 UTC)" (scheduled at 2023-01-17 05:11:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:11:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:11:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 48, 831252), True, datetime.datetime(2023, 1, 17, 5, 11, 48, 831252), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:11:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:11:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:11:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:11:52,625 [239] [DEBUG] [app] Starting request: urn:request:b9bdf59d-d5e3-45bc-bb06-7c371060e18c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:52,625 [243] [DEBUG] [app] Starting request: urn:request:783d6da8-c75a-4f23-848e-167a0c55bc0e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:11:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:52,630 [232] [DEBUG] [app] Starting request: urn:request:526bd096-1f78-4451-af0a-db90c428f9b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,630 [240] [DEBUG] [app] Starting request: urn:request:0c06b10e-4ecd-4342-a403-a6b762cc5460 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,631 [232] [DEBUG] [app] Ending request: urn:request:526bd096-1f78-4451-af0a-db90c428f9b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:526bd096-1f78-4451-af0a-db90c428f9b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:11:52,631 [240] [DEBUG] [app] Ending request: urn:request:0c06b10e-4ecd-4342-a403-a6b762cc5460 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c06b10e-4ecd-4342-a403-a6b762cc5460', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:11:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:11:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:11:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:52,636 [244] [DEBUG] [app] Starting request: urn:request:5395b709-c946-4e3a-be18-374c30fa61d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:52,636 [243] [DEBUG] [app] Starting request: urn:request:4f349030-cdcb-46bc-90a8-ccf25eca935b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:52,636 [244] [DEBUG] [app] Ending request: urn:request:5395b709-c946-4e3a-be18-374c30fa61d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5395b709-c946-4e3a-be18-374c30fa61d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:11:52,636 [243] [DEBUG] [app] Ending request: urn:request:4f349030-cdcb-46bc-90a8-ccf25eca935b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f349030-cdcb-46bc-90a8-ccf25eca935b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:52,648 [239] [DEBUG] [app] Ending request: urn:request:b9bdf59d-d5e3-45bc-bb06-7c371060e18c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b9bdf59d-d5e3-45bc-bb06-7c371060e18c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:52,648 [243] [DEBUG] [app] Ending request: urn:request:783d6da8-c75a-4f23-848e-167a0c55bc0e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:783d6da8-c75a-4f23-848e-167a0c55bc0e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:52,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:11:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:11:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:11:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:09.230038+00:00 (in 16.392202 seconds) securityworker stdout | 2023-01-17 05:11:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:22 UTC)" (scheduled at 2023-01-17 05:11:52.837365+00:00) securityworker stdout | 2023-01-17 05:11:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:11:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:11:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:27.226424+00:00 (in 29.999511 seconds) buildlogsarchiver stdout | 2023-01-17 05:11:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:27 UTC)" (scheduled at 2023-01-17 05:11:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:11:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 11, 57, 227267), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:11:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:11:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:11:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:27 UTC)" executed successfully builder stdout | 2023-01-17 05:11:58,012 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:58,021 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:58,021 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:58,027 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:08.735066+00:00 (in 9.999484 seconds) notificationworker stdout | 2023-01-17 05:11:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:08 UTC)" (scheduled at 2023-01-17 05:11:58.735066+00:00) notificationworker stdout | 2023-01-17 05:11:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 58, 736030), True, datetime.datetime(2023, 1, 17, 5, 11, 58, 736030), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:11:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:11:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:47.932490+00:00 (in 48.594655 seconds) namespacegcworker stdout | 2023-01-17 05:11:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:59 UTC)" (scheduled at 2023-01-17 05:11:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:11:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:11:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 59, 338219), True, datetime.datetime(2023, 1, 17, 5, 11, 59, 338219), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:11:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:11:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:11:59,345 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:59,431 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:11:59,657 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:11:59,941 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:12:00,718 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:12:00,742 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:12:01,915 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:12:02,771 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:12:02,774 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:12:03,191 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:12:04,041 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:12:04,150 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} repositorygcworker stdout | 2023-01-17 05:12:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:12:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:39.030814+00:00 (in 34.292334 seconds) repositorygcworker stdout | 2023-01-17 05:12:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:17:04 UTC)" (scheduled at 2023-01-17 05:12:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:12:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 12, 4, 738875), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:12:04,747 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 12, 4, 738875), True, datetime.datetime(2023, 1, 17, 5, 12, 4, 738875), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:12:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 12, 4, 738875), True, datetime.datetime(2023, 1, 17, 5, 12, 4, 738875), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 12, 4, 738875), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:12:04,751 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:12:04,751 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:17:04 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:12:05,136 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:12:05,635 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:12:05,639 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:12:05,641 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:12:05,801 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gcworker stdout | 2023-01-17 05:12:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:12:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:35.848675+00:00 (in 29.999511 seconds) gcworker stdout | 2023-01-17 05:12:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:35 UTC)" (scheduled at 2023-01-17 05:12:05.848675+00:00) gcworker stdout | 2023-01-17 05:12:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722725849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:12:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:12:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:12:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:35 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:12:05,912 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:12:07,072 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:12:07,626 [243] [DEBUG] [app] Starting request: urn:request:1afa7515-739d-4f10-a34f-586fed599808 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:07,626 [239] [DEBUG] [app] Starting request: urn:request:7c412a3d-2a9c-48cd-bf9a-cc2f48e75002 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:07,631 [232] [DEBUG] [app] Starting request: urn:request:2baf4485-40bb-47d9-972e-a78c541d3ea5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:07,631 [232] [DEBUG] [app] Ending request: urn:request:2baf4485-40bb-47d9-972e-a78c541d3ea5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2baf4485-40bb-47d9-972e-a78c541d3ea5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:07,631 [242] [DEBUG] [app] Starting request: urn:request:8954b416-22e7-40f2-a81f-5df65db9539d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:12:07,632 [242] [DEBUG] [app] Ending request: urn:request:8954b416-22e7-40f2-a81f-5df65db9539d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8954b416-22e7-40f2-a81f-5df65db9539d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:12:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:07,636 [239] [DEBUG] [app] Starting request: urn:request:c920429a-c174-446b-b4d8-6a86a9e17771 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:07,637 [244] [DEBUG] [app] Starting request: urn:request:6bbb345f-7c80-48fe-b694-223994927a37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:07,637 [239] [DEBUG] [app] Ending request: urn:request:c920429a-c174-446b-b4d8-6a86a9e17771 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c920429a-c174-446b-b4d8-6a86a9e17771', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:07,637 [244] [DEBUG] [app] Ending request: urn:request:6bbb345f-7c80-48fe-b694-223994927a37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bbb345f-7c80-48fe-b694-223994927a37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:12:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:12:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:07,647 [239] [DEBUG] [app] Ending request: urn:request:7c412a3d-2a9c-48cd-bf9a-cc2f48e75002 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c412a3d-2a9c-48cd-bf9a-cc2f48e75002', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:07,648 [243] [DEBUG] [app] Ending request: urn:request:1afa7515-739d-4f10-a34f-586fed599808 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1afa7515-739d-4f10-a34f-586fed599808', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:12:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) notificationworker stdout | 2023-01-17 05:12:07,676 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:12:07,958 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:12:07,961 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:12:07,963 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:12:07,966 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:12:07,968 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:12:07,971 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:12:07,975 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:12:07,977 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:12:07,980 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} builder stdout | 2023-01-17 05:12:08,032 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:08,040 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:08,040 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:08,046 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:08,226 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:12:08,229 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:12:08,237 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:12:08,240 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:12:08,242 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:12:08,407 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:12:08,475 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:12:08,494 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:12:08,573 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:12:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:18.735066+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:12:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:18 UTC)" (scheduled at 2023-01-17 05:12:08.735066+00:00) notificationworker stdout | 2023-01-17 05:12:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 8, 735916), True, datetime.datetime(2023, 1, 17, 5, 12, 8, 735916), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:22.837365+00:00 (in 13.606845 seconds) securityworker stdout | 2023-01-17 05:12:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:39 UTC)" (scheduled at 2023-01-17 05:12:09.230038+00:00) securityworker stdout | 2023-01-17 05:12:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:12:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:12:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:12:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:12:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:12:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:12:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:39 UTC)" executed successfully builder stdout | 2023-01-17 05:12:18,056 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:18,064 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:18,065 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:18,070 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:28.735066+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:12:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:28 UTC)" (scheduled at 2023-01-17 05:12:18.735066+00:00) notificationworker stdout | 2023-01-17 05:12:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 18, 735956), True, datetime.datetime(2023, 1, 17, 5, 12, 18, 735956), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:12:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:12:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:25.334138+00:00 (in 5.091448 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:12:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:20 UTC)" (scheduled at 2023-01-17 05:12:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:12:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:12:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 20, 243133), True, datetime.datetime(2023, 1, 17, 5, 12, 20, 243133), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:12:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:12:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:12:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:12:22,625 [239] [DEBUG] [app] Starting request: urn:request:11fdfe23-7183-41b9-b3e0-86b8368ebb18 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:22,626 [244] [DEBUG] [app] Starting request: urn:request:878bafb7-41ce-4762-89a0-7be13c8cd09c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:22,630 [242] [DEBUG] [app] Starting request: urn:request:07dfcc03-5a3c-4829-801f-8ec334531292 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,631 [232] [DEBUG] [app] Starting request: urn:request:52cccbf8-dae4-46e4-9ee6-6a10ea4cd2ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,631 [242] [DEBUG] [app] Ending request: urn:request:07dfcc03-5a3c-4829-801f-8ec334531292 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07dfcc03-5a3c-4829-801f-8ec334531292', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,631 [232] [DEBUG] [app] Ending request: urn:request:52cccbf8-dae4-46e4-9ee6-6a10ea4cd2ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:52cccbf8-dae4-46e4-9ee6-6a10ea4cd2ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:12:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:22,636 [243] [DEBUG] [app] Starting request: urn:request:09cc68fc-d866-4b8e-9461-0323efb2543e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:22,636 [241] [DEBUG] [app] Starting request: urn:request:bfa43b52-ad71-4335-addf-4c9a0c1d9388 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:22,636 [243] [DEBUG] [app] Ending request: urn:request:09cc68fc-d866-4b8e-9461-0323efb2543e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:09cc68fc-d866-4b8e-9461-0323efb2543e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:22,637 [241] [DEBUG] [app] Ending request: urn:request:bfa43b52-ad71-4335-addf-4c9a0c1d9388 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bfa43b52-ad71-4335-addf-4c9a0c1d9388', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:22,648 [239] [DEBUG] [app] Ending request: urn:request:11fdfe23-7183-41b9-b3e0-86b8368ebb18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11fdfe23-7183-41b9-b3e0-86b8368ebb18', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:22,648 [244] [DEBUG] [app] Ending request: urn:request:878bafb7-41ce-4762-89a0-7be13c8cd09c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:878bafb7-41ce-4762-89a0-7be13c8cd09c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:12:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:12:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:39.230038+00:00 (in 16.392246 seconds) securityworker stdout | 2023-01-17 05:12:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:52 UTC)" (scheduled at 2023-01-17 05:12:22.837365+00:00) securityworker stdout | 2023-01-17 05:12:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:12:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:12:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:48.830274+00:00 (in 24.187907 seconds) exportactionlogsworker stdout | 2023-01-17 05:12:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:24 UTC)" (scheduled at 2023-01-17 05:12:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:12:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:12:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:12:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:12:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:20.242198+00:00 (in 54.907572 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:12:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:25 UTC)" (scheduled at 2023-01-17 05:12:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:12:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:12:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:12:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:12:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:57.226424+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 05:12:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:57 UTC)" (scheduled at 2023-01-17 05:12:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:12:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 12, 27, 227249), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:12:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:12:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:12:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:57 UTC)" executed successfully builder stdout | 2023-01-17 05:12:28,074 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:28,083 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:28,083 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:28,089 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:38.735066+00:00 (in 9.999494 seconds) notificationworker stdout | 2023-01-17 05:12:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:38 UTC)" (scheduled at 2023-01-17 05:12:28.735066+00:00) notificationworker stdout | 2023-01-17 05:12:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 28, 736066), True, datetime.datetime(2023, 1, 17, 5, 12, 28, 736066), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:29,467 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:12:29,682 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:12:29,970 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:12:30,741 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:12:30,776 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:12:31,934 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:12:32,789 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:12:32,791 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:12:33,213 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:12:34,077 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:12:34,185 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:12:35,157 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:12:35,642 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:12:35,644 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:12:35,647 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} logrotateworker stdout | 2023-01-17 05:12:35,837 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gcworker stdout | 2023-01-17 05:12:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:12:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:05.848675+00:00 (in 29.999554 seconds) gcworker stdout | 2023-01-17 05:12:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:05 UTC)" (scheduled at 2023-01-17 05:12:35.848675+00:00) gcworker stdout | 2023-01-17 05:12:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722755849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:12:35,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:12:35,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:12:35,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:05 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:12:35,947 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:12:37,095 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:12:37,625 [239] [DEBUG] [app] Starting request: urn:request:31d42ac3-413a-4871-b22e-adfb1ded557c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:37,625 [244] [DEBUG] [app] Starting request: urn:request:e9cc1d10-82c0-465a-982c-e0f4dfa9e2d5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:37,630 [232] [DEBUG] [app] Starting request: urn:request:a29c880c-585d-45d4-a9e5-b3f1bba86834 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,631 [232] [DEBUG] [app] Ending request: urn:request:a29c880c-585d-45d4-a9e5-b3f1bba86834 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a29c880c-585d-45d4-a9e5-b3f1bba86834', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,631 [236] [DEBUG] [app] Starting request: urn:request:dbfaca47-ca3b-40e0-ad74-4d63fb109336 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:12:37,631 [236] [DEBUG] [app] Ending request: urn:request:dbfaca47-ca3b-40e0-ad74-4d63fb109336 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dbfaca47-ca3b-40e0-ad74-4d63fb109336', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:37,636 [241] [DEBUG] [app] Starting request: urn:request:cc8290e0-d9fb-49ff-91ae-add3b8ab2b78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:37,636 [243] [DEBUG] [app] Starting request: urn:request:590050ba-9f50-4aac-abac-f4c29452e870 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:37,636 [241] [DEBUG] [app] Ending request: urn:request:cc8290e0-d9fb-49ff-91ae-add3b8ab2b78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cc8290e0-d9fb-49ff-91ae-add3b8ab2b78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:37,637 [243] [DEBUG] [app] Ending request: urn:request:590050ba-9f50-4aac-abac-f4c29452e870 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:590050ba-9f50-4aac-abac-f4c29452e870', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:12:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:37,648 [244] [DEBUG] [app] Ending request: urn:request:e9cc1d10-82c0-465a-982c-e0f4dfa9e2d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e9cc1d10-82c0-465a-982c-e0f4dfa9e2d5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:37,648 [239] [DEBUG] [app] Ending request: urn:request:31d42ac3-413a-4871-b22e-adfb1ded557c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:31d42ac3-413a-4871-b22e-adfb1ded557c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:12:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:12:37,689 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:12:37,967 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:12:37,970 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:12:37,972 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:12:37,975 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:12:37,978 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:12:37,980 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:12:37,982 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:12:37,985 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:12:37,987 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} builder stdout | 2023-01-17 05:12:38,089 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:38,098 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:38,098 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:38,104 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:38,233 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:12:38,236 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:12:38,244 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:12:38,247 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:12:38,249 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:12:38,442 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:12:38,510 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:12:38,529 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:12:38,586 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:12:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:43.935629+00:00 (in 5.200055 seconds) notificationworker stdout | 2023-01-17 05:12:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:48 UTC)" (scheduled at 2023-01-17 05:12:38.735066+00:00) notificationworker stdout | 2023-01-17 05:12:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 38, 735988), True, datetime.datetime(2023, 1, 17, 5, 12, 38, 735988), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:12:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:12:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:48.738458+00:00 (in 9.707184 seconds) repositorygcworker stdout | 2023-01-17 05:12:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:39 UTC)" (scheduled at 2023-01-17 05:12:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:12:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:12:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 39, 31738), True, datetime.datetime(2023, 1, 17, 5, 12, 39, 31738), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:12:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:12:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:12:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:52.837365+00:00 (in 13.606852 seconds) securityworker stdout | 2023-01-17 05:12:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:09 UTC)" (scheduled at 2023-01-17 05:12:39.230038+00:00) securityworker stdout | 2023-01-17 05:12:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:12:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:12:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:12:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:12:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:48.735066+00:00 (in 4.798977 seconds) notificationworker stdout | 2023-01-17 05:12:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:43 UTC)" (scheduled at 2023-01-17 05:12:43.935629+00:00) notificationworker stdout | 2023-01-17 05:12:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:12:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:12:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:12:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:59.337413+00:00 (in 11.404462 seconds) namespacegcworker stdout | 2023-01-17 05:12:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:47 UTC)" (scheduled at 2023-01-17 05:12:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:12:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:12:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:47 UTC)" executed successfully builder stdout | 2023-01-17 05:12:48,114 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:48,123 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:48,123 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:48,128 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:58.735066+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:12:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:58 UTC)" (scheduled at 2023-01-17 05:12:48.735066+00:00) notificationworker stdout | 2023-01-17 05:12:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 48, 735928), True, datetime.datetime(2023, 1, 17, 5, 12, 48, 735928), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:12:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:12:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:39.030814+00:00 (in 50.291844 seconds) repositorygcworker stdout | 2023-01-17 05:12:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:48 UTC)" (scheduled at 2023-01-17 05:12:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:12:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:12:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:12:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:12:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:12:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:24.641816+00:00 (in 35.811004 seconds) exportactionlogsworker stdout | 2023-01-17 05:12:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:48 UTC)" (scheduled at 2023-01-17 05:12:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:12:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:12:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 48, 831170), True, datetime.datetime(2023, 1, 17, 5, 12, 48, 831170), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:12:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:12:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:12:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:12:52,625 [241] [DEBUG] [app] Starting request: urn:request:5d50b131-0fb2-488c-a0c5-8396d6b2a303 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:52,625 [239] [DEBUG] [app] Starting request: urn:request:217d6e0a-5392-478d-b30d-c78018924eed (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:12:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:52,630 [240] [DEBUG] [app] Starting request: urn:request:62517d66-5557-42f2-b8bc-e17c4a38f9ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:52,631 [237] [DEBUG] [app] Starting request: urn:request:e47b3e0c-b3db-4ea7-8868-effeccd2a9ca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:52,631 [240] [DEBUG] [app] Ending request: urn:request:62517d66-5557-42f2-b8bc-e17c4a38f9ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:62517d66-5557-42f2-b8bc-e17c4a38f9ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:12:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:12:52,631 [237] [DEBUG] [app] Ending request: urn:request:e47b3e0c-b3db-4ea7-8868-effeccd2a9ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e47b3e0c-b3db-4ea7-8868-effeccd2a9ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:12:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:12:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:52,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:52,636 [239] [DEBUG] [app] Starting request: urn:request:580989ae-e6fe-45bb-aea5-c7141dc72b4f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:52,636 [239] [DEBUG] [app] Ending request: urn:request:580989ae-e6fe-45bb-aea5-c7141dc72b4f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:580989ae-e6fe-45bb-aea5-c7141dc72b4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:52,637 [244] [DEBUG] [app] Starting request: urn:request:ec905c42-1d49-488c-8419-1a79d986adc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,637 [244] [DEBUG] [app] Ending request: urn:request:ec905c42-1d49-488c-8419-1a79d986adc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec905c42-1d49-488c-8419-1a79d986adc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:52,647 [239] [DEBUG] [app] Ending request: urn:request:217d6e0a-5392-478d-b30d-c78018924eed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:217d6e0a-5392-478d-b30d-c78018924eed', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:52,648 [241] [DEBUG] [app] Ending request: urn:request:5d50b131-0fb2-488c-a0c5-8396d6b2a303 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d50b131-0fb2-488c-a0c5-8396d6b2a303', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:12:52,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:12:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:12:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:09.230038+00:00 (in 16.392226 seconds) securityworker stdout | 2023-01-17 05:12:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:22 UTC)" (scheduled at 2023-01-17 05:12:52.837365+00:00) securityworker stdout | 2023-01-17 05:12:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:12:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:12:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:27.226424+00:00 (in 29.999533 seconds) buildlogsarchiver stdout | 2023-01-17 05:12:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:27 UTC)" (scheduled at 2023-01-17 05:12:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:12:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 12, 57, 227280), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:12:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:12:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:12:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:27 UTC)" executed successfully builder stdout | 2023-01-17 05:12:58,139 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:58,147 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:58,147 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:58,153 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:08.735066+00:00 (in 9.999508 seconds) notificationworker stdout | 2023-01-17 05:12:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:08 UTC)" (scheduled at 2023-01-17 05:12:58.735066+00:00) notificationworker stdout | 2023-01-17 05:12:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 58, 736022), True, datetime.datetime(2023, 1, 17, 5, 12, 58, 736022), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:12:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:12:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:47.932490+00:00 (in 48.594602 seconds) namespacegcworker stdout | 2023-01-17 05:12:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:59 UTC)" (scheduled at 2023-01-17 05:12:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:12:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:12:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 59, 338307), True, datetime.datetime(2023, 1, 17, 5, 12, 59, 338307), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:12:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:12:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:12:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:59,502 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:12:59,717 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:12:59,976 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:13:00,764 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:13:00,791 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:13:01,970 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:13:02,826 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:13:02,828 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:13:03,237 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:13:04,113 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:13:04,216 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:13:05,192 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:13:05,649 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:13:05,652 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:13:05,654 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:13:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:13:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:35.848675+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 05:13:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:35 UTC)" (scheduled at 2023-01-17 05:13:05.848675+00:00) gcworker stdout | 2023-01-17 05:13:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722785849, 1209600, True, 3, 500, 1, 0]) logrotateworker stdout | 2023-01-17 05:13:05,856 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gcworker stdout | 2023-01-17 05:13:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:13:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:13:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:35 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:13:05,982 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:13:07,129 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:13:07,626 [243] [DEBUG] [app] Starting request: urn:request:b2300b8a-3577-457d-957b-230018b1323a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:07,626 [239] [DEBUG] [app] Starting request: urn:request:a7f88769-da57-4806-bdcd-8fb6f5949689 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:07,631 [237] [DEBUG] [app] Starting request: urn:request:d3bc851d-d795-4919-8964-aa0d8bb65c5b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,631 [237] [DEBUG] [app] Ending request: urn:request:d3bc851d-d795-4919-8964-aa0d8bb65c5b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3bc851d-d795-4919-8964-aa0d8bb65c5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,631 [240] [DEBUG] [app] Starting request: urn:request:6ec4a95f-cbc0-4cb2-b908-022ae3a491f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:13:07,632 [240] [DEBUG] [app] Ending request: urn:request:6ec4a95f-cbc0-4cb2-b908-022ae3a491f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ec4a95f-cbc0-4cb2-b908-022ae3a491f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:13:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:07,636 [244] [DEBUG] [app] Starting request: urn:request:dfe08afa-bd13-44ef-a616-24e43a719af4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:07,636 [239] [DEBUG] [app] Starting request: urn:request:8765121c-fdef-4bfb-a7ed-d8daec11d84b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:07,637 [244] [DEBUG] [app] Ending request: urn:request:dfe08afa-bd13-44ef-a616-24e43a719af4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dfe08afa-bd13-44ef-a616-24e43a719af4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:07,637 [239] [DEBUG] [app] Ending request: urn:request:8765121c-fdef-4bfb-a7ed-d8daec11d84b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8765121c-fdef-4bfb-a7ed-d8daec11d84b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:13:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:13:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:07,648 [239] [DEBUG] [app] Ending request: urn:request:a7f88769-da57-4806-bdcd-8fb6f5949689 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7f88769-da57-4806-bdcd-8fb6f5949689', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:07,648 [243] [DEBUG] [app] Ending request: urn:request:b2300b8a-3577-457d-957b-230018b1323a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2300b8a-3577-457d-957b-230018b1323a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:13:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:13:07,703 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:13:07,977 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:13:07,980 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:13:07,983 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:13:07,986 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:13:07,988 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:13:07,990 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:13:07,993 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:13:07,995 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:13:07,998 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} builder stdout | 2023-01-17 05:13:08,163 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:08,172 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:08,172 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:08,178 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:08,241 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:13:08,243 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:13:08,249 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:13:08,253 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:13:08,255 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:13:08,473 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:13:08,545 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:13:08,558 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:13:08,621 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:13:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:18.735066+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:13:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:18 UTC)" (scheduled at 2023-01-17 05:13:08.735066+00:00) notificationworker stdout | 2023-01-17 05:13:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 8, 735959), True, datetime.datetime(2023, 1, 17, 5, 13, 8, 735959), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:22.837365+00:00 (in 13.606835 seconds) securityworker stdout | 2023-01-17 05:13:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:39 UTC)" (scheduled at 2023-01-17 05:13:09.230038+00:00) securityworker stdout | 2023-01-17 05:13:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:13:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:13:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:13:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:13:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:13:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:13:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:39 UTC)" executed successfully builder stdout | 2023-01-17 05:13:18,188 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:18,197 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:18,197 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:18,203 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:28.735066+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:13:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:28 UTC)" (scheduled at 2023-01-17 05:13:18.735066+00:00) notificationworker stdout | 2023-01-17 05:13:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 18, 735963), True, datetime.datetime(2023, 1, 17, 5, 13, 18, 735963), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:13:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:13:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:25.334138+00:00 (in 5.091463 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:13:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:20 UTC)" (scheduled at 2023-01-17 05:13:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:13:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:13:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 20, 243175), True, datetime.datetime(2023, 1, 17, 5, 13, 20, 243175), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:13:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:13:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:13:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:13:22,626 [239] [DEBUG] [app] Starting request: urn:request:7c3814d5-1674-4680-92f1-3fdc1b0a88ae (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:22,626 [244] [DEBUG] [app] Starting request: urn:request:3a22ef95-6443-40b3-8dd8-ae4dbc774713 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:22,631 [242] [DEBUG] [app] Starting request: urn:request:85c23a55-44c1-4048-98e1-b3e28f738882 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,631 [232] [DEBUG] [app] Starting request: urn:request:9548866d-eb5f-4a69-aa1c-655de6aba94b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,632 [232] [DEBUG] [app] Ending request: urn:request:9548866d-eb5f-4a69-aa1c-655de6aba94b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9548866d-eb5f-4a69-aa1c-655de6aba94b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,632 [242] [DEBUG] [app] Ending request: urn:request:85c23a55-44c1-4048-98e1-b3e28f738882 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85c23a55-44c1-4048-98e1-b3e28f738882', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:13:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:22,636 [243] [DEBUG] [app] Starting request: urn:request:dd9d1a8c-fcd0-42a8-9955-54ce9d677e2b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:22,637 [244] [DEBUG] [app] Starting request: urn:request:6644510f-86d2-4382-a5b2-4572d8eeeeab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:22,637 [243] [DEBUG] [app] Ending request: urn:request:dd9d1a8c-fcd0-42a8-9955-54ce9d677e2b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd9d1a8c-fcd0-42a8-9955-54ce9d677e2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:13:22,637 [244] [DEBUG] [app] Ending request: urn:request:6644510f-86d2-4382-a5b2-4572d8eeeeab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6644510f-86d2-4382-a5b2-4572d8eeeeab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:13:22,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:22,648 [244] [DEBUG] [app] Ending request: urn:request:3a22ef95-6443-40b3-8dd8-ae4dbc774713 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a22ef95-6443-40b3-8dd8-ae4dbc774713', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:22,649 [239] [DEBUG] [app] Ending request: urn:request:7c3814d5-1674-4680-92f1-3fdc1b0a88ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c3814d5-1674-4680-92f1-3fdc1b0a88ae', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:22,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:13:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) securityworker stdout | 2023-01-17 05:13:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:39.230038+00:00 (in 16.392232 seconds) securityworker stdout | 2023-01-17 05:13:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:52 UTC)" (scheduled at 2023-01-17 05:13:22.837365+00:00) securityworker stdout | 2023-01-17 05:13:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:22,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:13:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:13:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:48.830274+00:00 (in 24.188010 seconds) exportactionlogsworker stdout | 2023-01-17 05:13:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:24 UTC)" (scheduled at 2023-01-17 05:13:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:13:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:13:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:13:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:13:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:20.242198+00:00 (in 54.907493 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:13:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:25 UTC)" (scheduled at 2023-01-17 05:13:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:13:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:13:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:13:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:13:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:57.226424+00:00 (in 29.999506 seconds) buildlogsarchiver stdout | 2023-01-17 05:13:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:57 UTC)" (scheduled at 2023-01-17 05:13:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:13:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 13, 27, 227330), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:13:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:13:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:13:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:57 UTC)" executed successfully builder stdout | 2023-01-17 05:13:28,211 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:28,219 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:28,219 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:28,225 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:38.735066+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 05:13:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:38 UTC)" (scheduled at 2023-01-17 05:13:28.735066+00:00) notificationworker stdout | 2023-01-17 05:13:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 28, 736035), True, datetime.datetime(2023, 1, 17, 5, 13, 28, 736035), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:29,538 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:13:29,726 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:13:30,011 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:13:30,799 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:13:30,806 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:13:32,006 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:13:32,834 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:13:32,861 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:13:33,269 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:13:34,136 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:13:34,229 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:13:35,199 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:13:35,658 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:13:35,661 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:13:35,663 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:13:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:13:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:05.848675+00:00 (in 29.999444 seconds) gcworker stdout | 2023-01-17 05:13:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:05 UTC)" (scheduled at 2023-01-17 05:13:35.848675+00:00) gcworker stdout | 2023-01-17 05:13:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722815849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:13:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:13:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:13:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:13:35,892 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:13:35,995 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:13:37,161 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:13:37,626 [241] [DEBUG] [app] Starting request: urn:request:04a600bf-3c9c-46fc-aeb0-2678430ba65a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:37,626 [244] [DEBUG] [app] Starting request: urn:request:84ab79aa-a476-4227-a4e9-bdb8dbb50c6b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:37,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:37,631 [237] [DEBUG] [app] Starting request: urn:request:94c45a9c-4da4-4085-8e76-e7a099155cda (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:37,632 [237] [DEBUG] [app] Ending request: urn:request:94c45a9c-4da4-4085-8e76-e7a099155cda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:94c45a9c-4da4-4085-8e76-e7a099155cda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:37,632 [240] [DEBUG] [app] Starting request: urn:request:81bc9556-aa77-4732-bd15-c4f512a1c15e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:37,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:13:37,632 [240] [DEBUG] [app] Ending request: urn:request:81bc9556-aa77-4732-bd15-c4f512a1c15e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81bc9556-aa77-4732-bd15-c4f512a1c15e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:13:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:37,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:37,636 [239] [DEBUG] [app] Starting request: urn:request:3b103aeb-3a22-49cb-bb41-1278304625ac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:37,637 [243] [DEBUG] [app] Starting request: urn:request:a2ff936d-bb6a-46e3-b9ab-05ca470c4cd0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:37,637 [239] [DEBUG] [app] Ending request: urn:request:3b103aeb-3a22-49cb-bb41-1278304625ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b103aeb-3a22-49cb-bb41-1278304625ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:13:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:37,637 [243] [DEBUG] [app] Ending request: urn:request:a2ff936d-bb6a-46e3-b9ab-05ca470c4cd0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2ff936d-bb6a-46e3-b9ab-05ca470c4cd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:13:37,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:37,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:37,648 [244] [DEBUG] [app] Ending request: urn:request:84ab79aa-a476-4227-a4e9-bdb8dbb50c6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84ab79aa-a476-4227-a4e9-bdb8dbb50c6b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:37,648 [241] [DEBUG] [app] Ending request: urn:request:04a600bf-3c9c-46fc-aeb0-2678430ba65a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04a600bf-3c9c-46fc-aeb0-2678430ba65a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:37,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:13:37,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) notificationworker stdout | 2023-01-17 05:13:37,731 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:13:37,986 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:13:37,989 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:13:37,991 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:13:37,994 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:13:37,996 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:13:37,998 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:13:38,002 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:13:38,004 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:13:38,006 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} builder stdout | 2023-01-17 05:13:38,235 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:38,243 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:38,243 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-web stdout | 2023-01-17 05:13:38,247 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} builder stdout | 2023-01-17 05:13:38,250 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:38,250 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:13:38,255 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:13:38,259 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:13:38,262 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:13:38,482 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:13:38,580 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:13:38,585 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:13:38,644 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:13:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:43.935629+00:00 (in 5.200030 seconds) notificationworker stdout | 2023-01-17 05:13:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:48 UTC)" (scheduled at 2023-01-17 05:13:38.735066+00:00) notificationworker stdout | 2023-01-17 05:13:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 38, 736025), True, datetime.datetime(2023, 1, 17, 5, 13, 38, 736025), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:13:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:13:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:48.738458+00:00 (in 9.707177 seconds) repositorygcworker stdout | 2023-01-17 05:13:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:39 UTC)" (scheduled at 2023-01-17 05:13:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:13:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:13:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 39, 31696), True, datetime.datetime(2023, 1, 17, 5, 13, 39, 31696), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:13:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:13:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:13:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:52.837365+00:00 (in 13.606894 seconds) securityworker stdout | 2023-01-17 05:13:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:09 UTC)" (scheduled at 2023-01-17 05:13:39.230038+00:00) securityworker stdout | 2023-01-17 05:13:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:13:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:13:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:13:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:13:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:48.735066+00:00 (in 4.798953 seconds) notificationworker stdout | 2023-01-17 05:13:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:43 UTC)" (scheduled at 2023-01-17 05:13:43.935629+00:00) notificationworker stdout | 2023-01-17 05:13:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:13:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:13:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:13:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:59.337413+00:00 (in 11.404440 seconds) namespacegcworker stdout | 2023-01-17 05:13:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:47 UTC)" (scheduled at 2023-01-17 05:13:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:13:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:13:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:47 UTC)" executed successfully builder stdout | 2023-01-17 05:13:48,260 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:48,269 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:48,269 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:48,275 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:56.935055+00:00 (in 8.199486 seconds) notificationworker stdout | 2023-01-17 05:13:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:58 UTC)" (scheduled at 2023-01-17 05:13:48.735066+00:00) notificationworker stdout | 2023-01-17 05:13:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 48, 735990), True, datetime.datetime(2023, 1, 17, 5, 13, 48, 735990), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:13:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:13:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:39.030814+00:00 (in 50.291908 seconds) repositorygcworker stdout | 2023-01-17 05:13:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:48 UTC)" (scheduled at 2023-01-17 05:13:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:13:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:13:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:13:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:13:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:13:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:24.641816+00:00 (in 35.811057 seconds) exportactionlogsworker stdout | 2023-01-17 05:13:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:48 UTC)" (scheduled at 2023-01-17 05:13:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:13:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:13:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 48, 831157), True, datetime.datetime(2023, 1, 17, 5, 13, 48, 831157), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:13:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:13:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:13:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:13:52,625 [239] [DEBUG] [app] Starting request: urn:request:327bd919-3965-43b4-b40c-c80182c40ff8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:52,625 [241] [DEBUG] [app] Starting request: urn:request:2995aba2-c7f3-40b8-8850-b86365bcf14e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:13:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:52,630 [232] [DEBUG] [app] Starting request: urn:request:7ce83355-4137-4279-90b5-7792d6b4f9d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:52,631 [232] [DEBUG] [app] Ending request: urn:request:7ce83355-4137-4279-90b5-7792d6b4f9d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ce83355-4137-4279-90b5-7792d6b4f9d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:52,631 [236] [DEBUG] [app] Starting request: urn:request:7176fa0a-5c83-4038-9b41-12bf15a999ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:13:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:13:52,631 [236] [DEBUG] [app] Ending request: urn:request:7176fa0a-5c83-4038-9b41-12bf15a999ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7176fa0a-5c83-4038-9b41-12bf15a999ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:13:52,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:52,636 [244] [DEBUG] [app] Starting request: urn:request:a0f4a72a-0e17-405b-9c76-4f2bfdf479ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:52,636 [239] [DEBUG] [app] Starting request: urn:request:fbb941b7-fb71-4a6a-b8a8-91e031757257 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:52,636 [244] [DEBUG] [app] Ending request: urn:request:a0f4a72a-0e17-405b-9c76-4f2bfdf479ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0f4a72a-0e17-405b-9c76-4f2bfdf479ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:52,636 [239] [DEBUG] [app] Ending request: urn:request:fbb941b7-fb71-4a6a-b8a8-91e031757257 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fbb941b7-fb71-4a6a-b8a8-91e031757257', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:13:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:52,648 [239] [DEBUG] [app] Ending request: urn:request:327bd919-3965-43b4-b40c-c80182c40ff8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:327bd919-3965-43b4-b40c-c80182c40ff8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:52,648 [241] [DEBUG] [app] Ending request: urn:request:2995aba2-c7f3-40b8-8850-b86365bcf14e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2995aba2-c7f3-40b8-8850-b86365bcf14e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:13:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:13:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:09.230038+00:00 (in 16.392242 seconds) securityworker stdout | 2023-01-17 05:13:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:22 UTC)" (scheduled at 2023-01-17 05:13:52.837365+00:00) securityworker stdout | 2023-01-17 05:13:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:52,852 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:52,852 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:13:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:58.735066+00:00 (in 1.799529 seconds) notificationworker stdout | 2023-01-17 05:13:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:18:56 UTC)" (scheduled at 2023-01-17 05:13:56.935055+00:00) notificationworker stdout | 2023-01-17 05:13:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 13, 56, 935911), 'notification/%']) notificationworker stdout | 2023-01-17 05:13:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 13, 56, 935911), True, datetime.datetime(2023, 1, 17, 5, 13, 56, 935911), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:13:56,946 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 13, 56, 935911), True, datetime.datetime(2023, 1, 17, 5, 13, 56, 935911), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 13, 56, 935911), 'notification/%']) notificationworker stdout | 2023-01-17 05:13:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:56,948 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:18:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:13:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:13:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:27.226424+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 05:13:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:27 UTC)" (scheduled at 2023-01-17 05:13:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:13:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 13, 57, 227297), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:13:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:13:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:13:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:27 UTC)" executed successfully builder stdout | 2023-01-17 05:13:58,285 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:58,294 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:58,294 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:58,299 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:08.735066+00:00 (in 9.999568 seconds) notificationworker stdout | 2023-01-17 05:13:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:08 UTC)" (scheduled at 2023-01-17 05:13:58.735066+00:00) notificationworker stdout | 2023-01-17 05:13:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 58, 735957), True, datetime.datetime(2023, 1, 17, 5, 13, 58, 735957), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:13:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:13:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:15.931994+00:00 (in 16.594119 seconds) namespacegcworker stdout | 2023-01-17 05:13:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:59 UTC)" (scheduled at 2023-01-17 05:13:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:13:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:13:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 59, 338288), True, datetime.datetime(2023, 1, 17, 5, 13, 59, 338288), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:13:59,350 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:13:59,350 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:13:59,350 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:59,574 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:13:59,762 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:14:00,034 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:14:00,835 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:14:00,841 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:14:02,041 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:14:02,870 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:14:02,876 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:14:03,300 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:14:04,172 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:14:04,263 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:14:05,235 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:14:05,666 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:14:05,668 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:14:05,671 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:14:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:14:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:35.848675+00:00 (in 29.999487 seconds) gcworker stdout | 2023-01-17 05:14:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:35 UTC)" (scheduled at 2023-01-17 05:14:05.848675+00:00) gcworker stdout | 2023-01-17 05:14:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722845849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:14:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:14:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:14:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:14:05,902 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:14:06,010 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:14:07,196 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:14:07,625 [239] [DEBUG] [app] Starting request: urn:request:7c6054cd-2966-48eb-aee2-852fe70b33d2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:07,625 [241] [DEBUG] [app] Starting request: urn:request:e44e4186-aa69-4e06-a105-6d30571a6134 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:07,630 [236] [DEBUG] [app] Starting request: urn:request:35382767-4f26-4a55-b932-b220027b520d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,631 [232] [DEBUG] [app] Starting request: urn:request:0b3bc6d9-7fe0-4866-bb21-47fb06782f0d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,631 [236] [DEBUG] [app] Ending request: urn:request:35382767-4f26-4a55-b932-b220027b520d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35382767-4f26-4a55-b932-b220027b520d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:14:07,631 [232] [DEBUG] [app] Ending request: urn:request:0b3bc6d9-7fe0-4866-bb21-47fb06782f0d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b3bc6d9-7fe0-4866-bb21-47fb06782f0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:14:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:07,636 [244] [DEBUG] [app] Starting request: urn:request:114e52ca-0fd9-4643-8e1c-7367d6dcae7f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:07,636 [239] [DEBUG] [app] Starting request: urn:request:49e51576-a3a8-48da-98b6-d9304326ab04 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:07,636 [244] [DEBUG] [app] Ending request: urn:request:114e52ca-0fd9-4643-8e1c-7367d6dcae7f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:114e52ca-0fd9-4643-8e1c-7367d6dcae7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:07,636 [239] [DEBUG] [app] Ending request: urn:request:49e51576-a3a8-48da-98b6-d9304326ab04 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49e51576-a3a8-48da-98b6-d9304326ab04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:14:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:07,647 [239] [DEBUG] [app] Ending request: urn:request:7c6054cd-2966-48eb-aee2-852fe70b33d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c6054cd-2966-48eb-aee2-852fe70b33d2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:07,648 [241] [DEBUG] [app] Ending request: urn:request:e44e4186-aa69-4e06-a105-6d30571a6134 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e44e4186-aa69-4e06-a105-6d30571a6134', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:14:07,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:14:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:14:07,742 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:14:07,995 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:14:07,997 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:14:08,000 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:14:08,003 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:14:08,005 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:14:08,007 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:14:08,010 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:14:08,013 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:14:08,015 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:14:08,254 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:14:08,257 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:14:08,261 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:14:08,266 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:14:08,268 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:14:08,310 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:08,318 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:08,318 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:08,324 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:14:08,512 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:14:08,585 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:14:08,620 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:14:08,679 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:14:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:18.735066+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:14:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:18 UTC)" (scheduled at 2023-01-17 05:14:08.735066+00:00) notificationworker stdout | 2023-01-17 05:14:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 8, 735915), True, datetime.datetime(2023, 1, 17, 5, 14, 8, 735915), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:18 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,126 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,127 [64] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 05:14:09 UTC)" (scheduled at 2023-01-17 05:14:09.126527+00:00) expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,127 [64] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00 expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,128 [64] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2023, 1, 16, 5, 14, 9, 127850)]) expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,128 [64] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:09.126527+00:00 (in 3599.999655 seconds) expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,134 [64] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:14:09,134 [64] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 06:14:09 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:22.837365+00:00 (in 13.606903 seconds) securityworker stdout | 2023-01-17 05:14:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:39 UTC)" (scheduled at 2023-01-17 05:14:09.230038+00:00) securityworker stdout | 2023-01-17 05:14:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:14:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:14:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:09,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:14:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:14:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:14:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:14:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:39 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:14:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:14:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:47.932490+00:00 (in 31.999998 seconds) namespacegcworker stdout | 2023-01-17 05:14:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:19:15 UTC)" (scheduled at 2023-01-17 05:14:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:14:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 14, 15, 932848), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:14:15,939 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 14, 15, 932848), True, datetime.datetime(2023, 1, 17, 5, 14, 15, 932848), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:14:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 14, 15, 932848), True, datetime.datetime(2023, 1, 17, 5, 14, 15, 932848), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 14, 15, 932848), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:14:15,943 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:14:15,943 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:19:15 UTC)" executed successfully builder stdout | 2023-01-17 05:14:18,334 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:18,343 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:18,343 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:18,348 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:28.735066+00:00 (in 9.999537 seconds) notificationworker stdout | 2023-01-17 05:14:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:28 UTC)" (scheduled at 2023-01-17 05:14:18.735066+00:00) notificationworker stdout | 2023-01-17 05:14:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 18, 735960), True, datetime.datetime(2023, 1, 17, 5, 14, 18, 735960), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:14:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:14:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:25.334138+00:00 (in 5.091357 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:14:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:20 UTC)" (scheduled at 2023-01-17 05:14:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:14:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:14:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 20, 243232), True, datetime.datetime(2023, 1, 17, 5, 14, 20, 243232), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:14:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:14:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:14:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:14:22,625 [239] [DEBUG] [app] Starting request: urn:request:27af80e1-d959-439b-88ab-e6da5784fc04 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:22,626 [244] [DEBUG] [app] Starting request: urn:request:d8187275-41d0-4e90-af07-bda5720efcbf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:22,630 [232] [DEBUG] [app] Starting request: urn:request:a19befc5-8d2b-4bb3-a25f-44ee42bedeb8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:22,631 [237] [DEBUG] [app] Starting request: urn:request:a8e2669d-b339-4b7f-8dda-f9da3ca41b44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:22,631 [232] [DEBUG] [app] Ending request: urn:request:a19befc5-8d2b-4bb3-a25f-44ee42bedeb8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a19befc5-8d2b-4bb3-a25f-44ee42bedeb8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:14:22,631 [237] [DEBUG] [app] Ending request: urn:request:a8e2669d-b339-4b7f-8dda-f9da3ca41b44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a8e2669d-b339-4b7f-8dda-f9da3ca41b44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:14:22,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:14:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:22,635 [241] [DEBUG] [app] Starting request: urn:request:158dacad-962b-4846-a8f6-7f980dad612b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:22,636 [239] [DEBUG] [app] Starting request: urn:request:c9a65b40-64b6-404d-a526-4ccbe7c78daa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:22,636 [239] [DEBUG] [app] Ending request: urn:request:c9a65b40-64b6-404d-a526-4ccbe7c78daa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9a65b40-64b6-404d-a526-4ccbe7c78daa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:22,636 [241] [DEBUG] [app] Ending request: urn:request:158dacad-962b-4846-a8f6-7f980dad612b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:158dacad-962b-4846-a8f6-7f980dad612b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:22,648 [239] [DEBUG] [app] Ending request: urn:request:27af80e1-d959-439b-88ab-e6da5784fc04 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27af80e1-d959-439b-88ab-e6da5784fc04', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:22,648 [244] [DEBUG] [app] Ending request: urn:request:d8187275-41d0-4e90-af07-bda5720efcbf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8187275-41d0-4e90-af07-bda5720efcbf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:22,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:14:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:14:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:14:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:39.230038+00:00 (in 16.392205 seconds) securityworker stdout | 2023-01-17 05:14:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:52 UTC)" (scheduled at 2023-01-17 05:14:22.837365+00:00) securityworker stdout | 2023-01-17 05:14:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:14:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:14:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:48.830274+00:00 (in 24.188012 seconds) exportactionlogsworker stdout | 2023-01-17 05:14:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:24 UTC)" (scheduled at 2023-01-17 05:14:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:14:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:14:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:14:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:14:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:20.242198+00:00 (in 54.907510 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:14:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:25 UTC)" (scheduled at 2023-01-17 05:14:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:14:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:14:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:14:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:14:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:57.226424+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 05:14:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:57 UTC)" (scheduled at 2023-01-17 05:14:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:14:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 14, 27, 227284), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:14:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:14:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:14:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:57 UTC)" executed successfully builder stdout | 2023-01-17 05:14:28,359 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:28,367 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:28,367 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:28,373 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:38.735066+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:14:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:38 UTC)" (scheduled at 2023-01-17 05:14:28.735066+00:00) notificationworker stdout | 2023-01-17 05:14:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 28, 736030), True, datetime.datetime(2023, 1, 17, 5, 14, 28, 736030), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:29,585 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:14:29,769 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:14:30,063 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:14:30,844 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:14:30,868 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:14:32,076 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:14:32,893 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:14:32,910 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:14:33,317 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:14:34,188 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:14:34,294 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:14:35,260 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:14:35,675 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:14:35,677 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:14:35,680 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:14:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:14:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:05.848675+00:00 (in 29.999537 seconds) gcworker stdout | 2023-01-17 05:14:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:05 UTC)" (scheduled at 2023-01-17 05:14:35.848675+00:00) gcworker stdout | 2023-01-17 05:14:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722875849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:14:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:14:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:14:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:14:35,928 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:14:36,045 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:14:37,205 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:14:37,625 [239] [DEBUG] [app] Starting request: urn:request:bdcc078d-4f7e-4531-90cb-99763ea3f986 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:37,626 [241] [DEBUG] [app] Starting request: urn:request:ce8c57e8-fc77-483c-84a8-2869f050defd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:37,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:37,631 [232] [DEBUG] [app] Starting request: urn:request:585fe6f2-b47c-448d-ba79-47bc471b0b82 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:37,631 [237] [DEBUG] [app] Starting request: urn:request:77f2673e-0398-4d19-ac31-bd1b6c5d88fc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:37,631 [232] [DEBUG] [app] Ending request: urn:request:585fe6f2-b47c-448d-ba79-47bc471b0b82 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:585fe6f2-b47c-448d-ba79-47bc471b0b82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:14:37,632 [237] [DEBUG] [app] Ending request: urn:request:77f2673e-0398-4d19-ac31-bd1b6c5d88fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77f2673e-0398-4d19-ac31-bd1b6c5d88fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:14:37,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:14:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:37,636 [243] [DEBUG] [app] Starting request: urn:request:7cd38c4c-74a5-4649-bd9a-9f4888702658 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:37,637 [244] [DEBUG] [app] Starting request: urn:request:8479966b-01d5-4c8a-9365-4efa0369c7ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:37,637 [243] [DEBUG] [app] Ending request: urn:request:7cd38c4c-74a5-4649-bd9a-9f4888702658 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7cd38c4c-74a5-4649-bd9a-9f4888702658', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:37,637 [244] [DEBUG] [app] Ending request: urn:request:8479966b-01d5-4c8a-9365-4efa0369c7ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8479966b-01d5-4c8a-9365-4efa0369c7ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:14:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:37,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:37,650 [241] [DEBUG] [app] Ending request: urn:request:ce8c57e8-fc77-483c-84a8-2869f050defd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce8c57e8-fc77-483c-84a8-2869f050defd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:37,650 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:37,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) gunicorn-web stdout | 2023-01-17 05:14:37,654 [239] [DEBUG] [app] Ending request: urn:request:bdcc078d-4f7e-4531-90cb-99763ea3f986 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bdcc078d-4f7e-4531-90cb-99763ea3f986', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:37,655 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:14:37,655 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:14:37,778 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:14:38,004 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:14:38,006 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:14:38,009 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:14:38,011 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:14:38,016 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:14:38,018 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:14:38,020 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:14:38,023 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:14:38,026 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:14:38,261 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:14:38,264 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:14:38,267 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:14:38,271 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:14:38,274 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:14:38,374 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:38,383 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:38,383 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:38,389 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:14:38,539 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:14:38,619 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:14:38,654 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:14:38,714 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:14:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:43.935629+00:00 (in 5.200068 seconds) notificationworker stdout | 2023-01-17 05:14:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:48 UTC)" (scheduled at 2023-01-17 05:14:38.735066+00:00) notificationworker stdout | 2023-01-17 05:14:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 38, 735955), True, datetime.datetime(2023, 1, 17, 5, 14, 38, 735955), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:14:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:14:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:48.738458+00:00 (in 9.707205 seconds) repositorygcworker stdout | 2023-01-17 05:14:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:39 UTC)" (scheduled at 2023-01-17 05:14:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:14:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:14:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 39, 31753), True, datetime.datetime(2023, 1, 17, 5, 14, 39, 31753), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:14:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:14:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:14:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:52.837365+00:00 (in 13.606865 seconds) securityworker stdout | 2023-01-17 05:14:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:09 UTC)" (scheduled at 2023-01-17 05:14:39.230038+00:00) securityworker stdout | 2023-01-17 05:14:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:14:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:14:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:14:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:14:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:14:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:14:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:14:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:48.735066+00:00 (in 4.798989 seconds) notificationworker stdout | 2023-01-17 05:14:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:43 UTC)" (scheduled at 2023-01-17 05:14:43.935629+00:00) notificationworker stdout | 2023-01-17 05:14:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:14:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:14:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:14:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:59.337413+00:00 (in 11.404472 seconds) namespacegcworker stdout | 2023-01-17 05:14:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:47 UTC)" (scheduled at 2023-01-17 05:14:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:14:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:14:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:47 UTC)" executed successfully builder stdout | 2023-01-17 05:14:48,399 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:48,407 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:48,407 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:48,413 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:58.735066+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 05:14:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:58 UTC)" (scheduled at 2023-01-17 05:14:48.735066+00:00) notificationworker stdout | 2023-01-17 05:14:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 48, 735949), True, datetime.datetime(2023, 1, 17, 5, 14, 48, 735949), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:14:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:14:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:39.030814+00:00 (in 50.291887 seconds) repositorygcworker stdout | 2023-01-17 05:14:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:48 UTC)" (scheduled at 2023-01-17 05:14:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:14:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:14:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:14:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:14:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:14:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:24.641816+00:00 (in 35.811109 seconds) exportactionlogsworker stdout | 2023-01-17 05:14:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:48 UTC)" (scheduled at 2023-01-17 05:14:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:14:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:14:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 48, 831062), True, datetime.datetime(2023, 1, 17, 5, 14, 48, 831062), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:14:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:14:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:14:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:14:52,625 [243] [DEBUG] [app] Starting request: urn:request:81b6e4b3-6a07-42d8-a705-e5c4b55881ce (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:52,625 [241] [DEBUG] [app] Starting request: urn:request:3a2aae8e-df17-46c3-ad87-9f03cbbf670a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:14:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:52,630 [242] [DEBUG] [app] Starting request: urn:request:69307dda-e9b3-438a-b3de-f53a6554b193 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,631 [237] [DEBUG] [app] Starting request: urn:request:17f9c0a8-2d4b-4c45-9634-ee3091f04846 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,631 [242] [DEBUG] [app] Ending request: urn:request:69307dda-e9b3-438a-b3de-f53a6554b193 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69307dda-e9b3-438a-b3de-f53a6554b193', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,631 [237] [DEBUG] [app] Ending request: urn:request:17f9c0a8-2d4b-4c45-9634-ee3091f04846 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:17f9c0a8-2d4b-4c45-9634-ee3091f04846', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:14:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:14:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:52,636 [244] [DEBUG] [app] Starting request: urn:request:a1e94556-5c71-4526-b922-f1438a467a10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:52,636 [241] [DEBUG] [app] Starting request: urn:request:8ac7fe3d-593e-4518-bcfb-0ab7587f1203 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:52,637 [244] [DEBUG] [app] Ending request: urn:request:a1e94556-5c71-4526-b922-f1438a467a10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1e94556-5c71-4526-b922-f1438a467a10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:52,637 [241] [DEBUG] [app] Ending request: urn:request:8ac7fe3d-593e-4518-bcfb-0ab7587f1203 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ac7fe3d-593e-4518-bcfb-0ab7587f1203', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:14:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:52,648 [241] [DEBUG] [app] Ending request: urn:request:3a2aae8e-df17-46c3-ad87-9f03cbbf670a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a2aae8e-df17-46c3-ad87-9f03cbbf670a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:52,648 [243] [DEBUG] [app] Ending request: urn:request:81b6e4b3-6a07-42d8-a705-e5c4b55881ce (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81b6e4b3-6a07-42d8-a705-e5c4b55881ce', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:52,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:52,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 05:14:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:14:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:09.230038+00:00 (in 16.392162 seconds) securityworker stdout | 2023-01-17 05:14:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:22 UTC)" (scheduled at 2023-01-17 05:14:52.837365+00:00) securityworker stdout | 2023-01-17 05:14:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:14:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:14:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:27.226424+00:00 (in 29.999510 seconds) buildlogsarchiver stdout | 2023-01-17 05:14:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:27 UTC)" (scheduled at 2023-01-17 05:14:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:14:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 14, 57, 227292), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:14:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:14:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:14:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:27 UTC)" executed successfully builder stdout | 2023-01-17 05:14:58,423 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:58,432 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:58,432 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:58,438 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:08.735066+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:14:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:08 UTC)" (scheduled at 2023-01-17 05:14:58.735066+00:00) notificationworker stdout | 2023-01-17 05:14:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 58, 735969), True, datetime.datetime(2023, 1, 17, 5, 14, 58, 735969), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:14:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:14:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:47.932490+00:00 (in 48.594656 seconds) namespacegcworker stdout | 2023-01-17 05:14:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:59 UTC)" (scheduled at 2023-01-17 05:14:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:14:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:14:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 59, 338229), True, datetime.datetime(2023, 1, 17, 5, 14, 59, 338229), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:14:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:14:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:14:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:59,594 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:14:59,804 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:15:00,099 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:15:00,874 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:15:00,879 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:15:02,103 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:15:02,899 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:15:02,920 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:15:03,352 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:15:04,209 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:15:04,329 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:15:05,267 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:15:05,683 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:15:05,686 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:15:05,689 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:15:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:15:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:35.848675+00:00 (in 29.999504 seconds) gcworker stdout | 2023-01-17 05:15:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:35 UTC)" (scheduled at 2023-01-17 05:15:05.848675+00:00) gcworker stdout | 2023-01-17 05:15:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722905849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:15:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:15:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:15:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:15:05,963 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:15:06,080 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:15:07,242 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:15:07,626 [241] [DEBUG] [app] Starting request: urn:request:417f38f5-93ec-4407-8b30-a9c6cf3ed052 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:07,626 [244] [DEBUG] [app] Starting request: urn:request:c34a23c4-313b-4c55-9af7-1d3a0e626f82 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:07,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:07,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:07,632 [232] [DEBUG] [app] Starting request: urn:request:d646c55c-31d5-4b8a-9d7b-867e3d832d4b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:07,632 [240] [DEBUG] [app] Starting request: urn:request:b177844f-8401-45f5-86d3-7268f5333dc3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:07,632 [232] [DEBUG] [app] Ending request: urn:request:d646c55c-31d5-4b8a-9d7b-867e3d832d4b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d646c55c-31d5-4b8a-9d7b-867e3d832d4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:15:07,633 [240] [DEBUG] [app] Ending request: urn:request:b177844f-8401-45f5-86d3-7268f5333dc3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b177844f-8401-45f5-86d3-7268f5333dc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:07,633 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:15:07,633 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:15:07,633 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:07,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,635 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,637 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:07,637 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:07,637 [239] [DEBUG] [app] Starting request: urn:request:72f2e061-19c5-4194-963d-7e5b72ef43eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:07,638 [239] [DEBUG] [app] Ending request: urn:request:72f2e061-19c5-4194-963d-7e5b72ef43eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72f2e061-19c5-4194-963d-7e5b72ef43eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:07,638 [243] [DEBUG] [app] Starting request: urn:request:c54c79d3-d230-4259-b759-5c1d66bfd86d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:07,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:15:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:07,639 [243] [DEBUG] [app] Ending request: urn:request:c54c79d3-d230-4259-b759-5c1d66bfd86d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c54c79d3-d230-4259-b759-5c1d66bfd86d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:07,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:07,639 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:07,639 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:15:07,640 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:07,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:07,644 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:07,650 [244] [DEBUG] [app] Ending request: urn:request:c34a23c4-313b-4c55-9af7-1d3a0e626f82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c34a23c4-313b-4c55-9af7-1d3a0e626f82', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:07,650 [241] [DEBUG] [app] Ending request: urn:request:417f38f5-93ec-4407-8b30-a9c6cf3ed052 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:417f38f5-93ec-4407-8b30-a9c6cf3ed052', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:07,650 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:07,650 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) gunicorn-web stdout | 2023-01-17 05:15:07,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:15:07,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:15:07,792 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:15:08,013 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:15:08,016 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:15:08,018 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:15:08,021 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:15:08,023 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:15:08,028 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:15:08,030 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:15:08,032 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:15:08,034 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:15:08,268 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:15:08,270 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:15:08,273 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:15:08,278 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:15:08,280 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:15:08,448 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:08,457 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:08,457 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:08,463 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:15:08,573 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:15:08,645 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:15:08,689 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:15:08,722 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:15:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:18.735066+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 05:15:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:18 UTC)" (scheduled at 2023-01-17 05:15:08.735066+00:00) notificationworker stdout | 2023-01-17 05:15:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 8, 735881), True, datetime.datetime(2023, 1, 17, 5, 15, 8, 735881), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:22.837365+00:00 (in 13.606830 seconds) securityworker stdout | 2023-01-17 05:15:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:39 UTC)" (scheduled at 2023-01-17 05:15:09.230038+00:00) securityworker stdout | 2023-01-17 05:15:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:15:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:15:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:09,245 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:09,245 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:15:09,246 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:39 UTC)" executed successfully builder stdout | 2023-01-17 05:15:18,473 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:18,482 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:18,482 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:18,488 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:28.735066+00:00 (in 9.999582 seconds) notificationworker stdout | 2023-01-17 05:15:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:28 UTC)" (scheduled at 2023-01-17 05:15:18.735066+00:00) notificationworker stdout | 2023-01-17 05:15:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 18, 735869), True, datetime.datetime(2023, 1, 17, 5, 15, 18, 735869), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:15:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:15:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:25.334138+00:00 (in 5.091384 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:15:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:20 UTC)" (scheduled at 2023-01-17 05:15:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:15:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:15:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 20, 243160), True, datetime.datetime(2023, 1, 17, 5, 15, 20, 243160), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:15:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:15:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:15:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:15:22,625 [241] [DEBUG] [app] Starting request: urn:request:f219f73a-a51b-4e56-b32c-829837f13d0e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:22,625 [239] [DEBUG] [app] Starting request: urn:request:549e7a25-9eec-47ca-b3bd-ced73dbcabee (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:22,630 [232] [DEBUG] [app] Starting request: urn:request:04c3605a-ac62-4ee6-8978-70cc7a2941dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:22,630 [242] [DEBUG] [app] Starting request: urn:request:93156a28-7fb4-4df4-9b8d-42e2e00086f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:22,630 [232] [DEBUG] [app] Ending request: urn:request:04c3605a-ac62-4ee6-8978-70cc7a2941dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04c3605a-ac62-4ee6-8978-70cc7a2941dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:15:22,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:15:22,630 [242] [DEBUG] [app] Ending request: urn:request:93156a28-7fb4-4df4-9b8d-42e2e00086f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93156a28-7fb4-4df4-9b8d-42e2e00086f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:15:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:22,635 [244] [DEBUG] [app] Starting request: urn:request:3fea9d02-37ba-4f86-884e-084581d84f7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:22,635 [244] [DEBUG] [app] Ending request: urn:request:3fea9d02-37ba-4f86-884e-084581d84f7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3fea9d02-37ba-4f86-884e-084581d84f7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:22,636 [239] [DEBUG] [app] Starting request: urn:request:3d042fcc-4471-4961-973a-ff795c2f6f63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:22,636 [239] [DEBUG] [app] Ending request: urn:request:3d042fcc-4471-4961-973a-ff795c2f6f63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d042fcc-4471-4961-973a-ff795c2f6f63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:22,646 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:22,648 [241] [DEBUG] [app] Ending request: urn:request:f219f73a-a51b-4e56-b32c-829837f13d0e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f219f73a-a51b-4e56-b32c-829837f13d0e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:22,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 05:15:22,651 [239] [DEBUG] [app] Ending request: urn:request:549e7a25-9eec-47ca-b3bd-ced73dbcabee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:549e7a25-9eec-47ca-b3bd-ced73dbcabee', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:22,651 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:22,651 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.027) securityworker stdout | 2023-01-17 05:15:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:39.230038+00:00 (in 16.392216 seconds) securityworker stdout | 2023-01-17 05:15:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:52 UTC)" (scheduled at 2023-01-17 05:15:22.837365+00:00) securityworker stdout | 2023-01-17 05:15:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:15:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:15:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:48.830274+00:00 (in 24.188011 seconds) exportactionlogsworker stdout | 2023-01-17 05:15:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:24 UTC)" (scheduled at 2023-01-17 05:15:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:15:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:15:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:15:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:15:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:20.242198+00:00 (in 54.907573 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:15:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:25 UTC)" (scheduled at 2023-01-17 05:15:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:15:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:15:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:15:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:15:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:57.226424+00:00 (in 29.999552 seconds) buildlogsarchiver stdout | 2023-01-17 05:15:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:57 UTC)" (scheduled at 2023-01-17 05:15:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:15:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 15, 27, 227246), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:15:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:15:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:15:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:57 UTC)" executed successfully builder stdout | 2023-01-17 05:15:28,498 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:28,507 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:28,507 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:28,512 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:38.735066+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 05:15:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:38 UTC)" (scheduled at 2023-01-17 05:15:28.735066+00:00) notificationworker stdout | 2023-01-17 05:15:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 28, 736040), True, datetime.datetime(2023, 1, 17, 5, 15, 28, 736040), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:29,630 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:15:29,839 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:15:30,110 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:15:30,906 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:15:30,909 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:15:32,117 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:15:32,934 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:15:32,936 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:15:33,362 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:15:34,244 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:15:34,360 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:15:35,279 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:15:35,690 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:15:35,693 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:15:35,695 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:15:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:15:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:05.848675+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:15:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:05 UTC)" (scheduled at 2023-01-17 05:15:35.848675+00:00) gcworker stdout | 2023-01-17 05:15:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722935849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:15:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:15:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:15:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:15:35,999 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:15:36,114 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:15:37,247 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:15:37,625 [239] [DEBUG] [app] Starting request: urn:request:c867323e-22ed-42af-88f0-096ffe562a8c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:37,626 [244] [DEBUG] [app] Starting request: urn:request:032c624e-8c39-4ebe-87c2-3cd3bcff679b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:37,631 [232] [DEBUG] [app] Starting request: urn:request:fa9f4a55-9844-4497-8d77-a5a48617866b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:37,631 [237] [DEBUG] [app] Starting request: urn:request:73b60a76-6bcb-4d69-936b-204491d2aef3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:37,631 [232] [DEBUG] [app] Ending request: urn:request:fa9f4a55-9844-4497-8d77-a5a48617866b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fa9f4a55-9844-4497-8d77-a5a48617866b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:15:37,632 [237] [DEBUG] [app] Ending request: urn:request:73b60a76-6bcb-4d69-936b-204491d2aef3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:73b60a76-6bcb-4d69-936b-204491d2aef3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:15:37,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:15:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:37,637 [241] [DEBUG] [app] Starting request: urn:request:2fbb03af-ea0b-4976-a33e-f8a612859603 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:37,637 [243] [DEBUG] [app] Starting request: urn:request:cba4b02b-cd3c-4e4c-ba5a-0e9d985e16c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:37,637 [241] [DEBUG] [app] Ending request: urn:request:2fbb03af-ea0b-4976-a33e-f8a612859603 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2fbb03af-ea0b-4976-a33e-f8a612859603', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:15:37,638 [243] [DEBUG] [app] Ending request: urn:request:cba4b02b-cd3c-4e4c-ba5a-0e9d985e16c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cba4b02b-cd3c-4e4c-ba5a-0e9d985e16c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:37,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:37,648 [244] [DEBUG] [app] Ending request: urn:request:032c624e-8c39-4ebe-87c2-3cd3bcff679b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:032c624e-8c39-4ebe-87c2-3cd3bcff679b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:37,648 [239] [DEBUG] [app] Ending request: urn:request:c867323e-22ed-42af-88f0-096ffe562a8c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c867323e-22ed-42af-88f0-096ffe562a8c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:15:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) notificationworker stdout | 2023-01-17 05:15:37,807 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:15:38,020 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:15:38,023 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:15:38,026 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:15:38,029 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:15:38,032 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:15:38,035 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:15:38,037 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:15:38,040 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:15:38,042 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:15:38,275 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:15:38,277 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:15:38,280 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:15:38,284 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:15:38,287 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:15:38,523 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:38,531 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:38,531 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:38,537 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:15:38,608 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:15:38,654 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:15:38,724 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:15:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:43.935629+00:00 (in 5.200056 seconds) notificationworker stdout | 2023-01-17 05:15:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:48 UTC)" (scheduled at 2023-01-17 05:15:38.735066+00:00) notificationworker stdout | 2023-01-17 05:15:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:38,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 38, 736045), True, datetime.datetime(2023, 1, 17, 5, 15, 38, 736045), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:48 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:15:38,746 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} repositorygcworker stdout | 2023-01-17 05:15:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:15:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:48.738458+00:00 (in 9.707169 seconds) repositorygcworker stdout | 2023-01-17 05:15:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:39 UTC)" (scheduled at 2023-01-17 05:15:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:15:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:15:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 39, 31706), True, datetime.datetime(2023, 1, 17, 5, 15, 39, 31706), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:15:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:15:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:15:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:52.837365+00:00 (in 13.606868 seconds) securityworker stdout | 2023-01-17 05:15:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:09 UTC)" (scheduled at 2023-01-17 05:15:39.230038+00:00) securityworker stdout | 2023-01-17 05:15:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:15:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:15:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:15:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:15:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:48.735066+00:00 (in 4.798961 seconds) notificationworker stdout | 2023-01-17 05:15:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:43 UTC)" (scheduled at 2023-01-17 05:15:43.935629+00:00) notificationworker stdout | 2023-01-17 05:15:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:15:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:15:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:15:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:59.337413+00:00 (in 11.404463 seconds) namespacegcworker stdout | 2023-01-17 05:15:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:47 UTC)" (scheduled at 2023-01-17 05:15:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:15:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:15:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:47 UTC)" executed successfully builder stdout | 2023-01-17 05:15:48,547 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:48,556 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:48,556 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:48,562 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:58.735066+00:00 (in 9.999476 seconds) notificationworker stdout | 2023-01-17 05:15:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:58 UTC)" (scheduled at 2023-01-17 05:15:48.735066+00:00) notificationworker stdout | 2023-01-17 05:15:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 48, 736013), True, datetime.datetime(2023, 1, 17, 5, 15, 48, 736013), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:15:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:15:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:39.030814+00:00 (in 50.291877 seconds) repositorygcworker stdout | 2023-01-17 05:15:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:48 UTC)" (scheduled at 2023-01-17 05:15:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:15:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:15:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:15:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:15:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:15:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:24.641816+00:00 (in 35.811093 seconds) exportactionlogsworker stdout | 2023-01-17 05:15:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:48 UTC)" (scheduled at 2023-01-17 05:15:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:15:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:15:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 48, 831151), True, datetime.datetime(2023, 1, 17, 5, 15, 48, 831151), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:15:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:15:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:15:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:15:52,626 [244] [DEBUG] [app] Starting request: urn:request:c41d5cfb-6a35-4ec3-8d17-a4e5336b6715 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:52,626 [239] [DEBUG] [app] Starting request: urn:request:bee2ec5a-88a9-4ebf-b80a-8cd9f3600506 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:15:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:52,631 [242] [DEBUG] [app] Starting request: urn:request:bba0e61b-8bd0-4454-97e1-4245c6c69cee (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,631 [235] [DEBUG] [app] Starting request: urn:request:4c162a96-53e9-4192-9b83-246c3ef8047d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,631 [242] [DEBUG] [app] Ending request: urn:request:bba0e61b-8bd0-4454-97e1-4245c6c69cee (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bba0e61b-8bd0-4454-97e1-4245c6c69cee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,631 [235] [DEBUG] [app] Ending request: urn:request:4c162a96-53e9-4192-9b83-246c3ef8047d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c162a96-53e9-4192-9b83-246c3ef8047d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:15:52,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:52,636 [244] [DEBUG] [app] Starting request: urn:request:9f1b191c-8757-4deb-ae3e-7cd11c23422e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:52,636 [244] [DEBUG] [app] Ending request: urn:request:9f1b191c-8757-4deb-ae3e-7cd11c23422e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f1b191c-8757-4deb-ae3e-7cd11c23422e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:52,637 [243] [DEBUG] [app] Starting request: urn:request:c8ee6d16-0b39-4090-950c-7893b2895322 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:52,637 [243] [DEBUG] [app] Ending request: urn:request:c8ee6d16-0b39-4090-950c-7893b2895322 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8ee6d16-0b39-4090-950c-7893b2895322', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:52,648 [244] [DEBUG] [app] Ending request: urn:request:c41d5cfb-6a35-4ec3-8d17-a4e5336b6715 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c41d5cfb-6a35-4ec3-8d17-a4e5336b6715', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:52,648 [239] [DEBUG] [app] Ending request: urn:request:bee2ec5a-88a9-4ebf-b80a-8cd9f3600506 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bee2ec5a-88a9-4ebf-b80a-8cd9f3600506', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:15:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:15:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:09.230038+00:00 (in 16.392206 seconds) securityworker stdout | 2023-01-17 05:15:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:22 UTC)" (scheduled at 2023-01-17 05:15:52.837365+00:00) securityworker stdout | 2023-01-17 05:15:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:15:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:15:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:27.226424+00:00 (in 29.999520 seconds) buildlogsarchiver stdout | 2023-01-17 05:15:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:27 UTC)" (scheduled at 2023-01-17 05:15:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:15:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 15, 57, 227294), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:15:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:15:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:15:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:27 UTC)" executed successfully builder stdout | 2023-01-17 05:15:58,563 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:58,573 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:58,573 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:58,579 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:08.735066+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:15:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:08 UTC)" (scheduled at 2023-01-17 05:15:58.735066+00:00) notificationworker stdout | 2023-01-17 05:15:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 58, 736035), True, datetime.datetime(2023, 1, 17, 5, 15, 58, 736035), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:15:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:15:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:47.932490+00:00 (in 48.594639 seconds) namespacegcworker stdout | 2023-01-17 05:15:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:59 UTC)" (scheduled at 2023-01-17 05:15:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:15:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:15:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 59, 338296), True, datetime.datetime(2023, 1, 17, 5, 15, 59, 338296), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:15:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:15:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:15:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:59,666 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:15:59,874 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:16:00,146 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:16:00,933 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:16:00,943 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:16:02,147 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:16:02,970 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:16:02,972 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:16:03,396 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:16:04,276 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:16:04,374 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:16:05,307 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:16:05,698 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:16:05,701 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:16:05,704 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:16:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:16:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:35.848675+00:00 (in 29.999495 seconds) gcworker stdout | 2023-01-17 05:16:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:35 UTC)" (scheduled at 2023-01-17 05:16:05.848675+00:00) gcworker stdout | 2023-01-17 05:16:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722965849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:16:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:16:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:16:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:16:06,034 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:16:06,149 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:16:07,283 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:16:07,625 [241] [DEBUG] [app] Starting request: urn:request:a3084a9e-02f5-493e-8f2d-10861e5169ae (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:07,625 [239] [DEBUG] [app] Starting request: urn:request:5374d3c9-8ec9-42b7-ae3d-3afd8efcb905 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:07,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:07,630 [242] [DEBUG] [app] Starting request: urn:request:09548af9-3c7b-4406-a630-d441078867b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,631 [242] [DEBUG] [app] Ending request: urn:request:09548af9-3c7b-4406-a630-d441078867b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09548af9-3c7b-4406-a630-d441078867b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,631 [240] [DEBUG] [app] Starting request: urn:request:9b007387-6e38-46f4-b1f6-43d948fd932d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:16:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:16:07,632 [240] [DEBUG] [app] Ending request: urn:request:9b007387-6e38-46f4-b1f6-43d948fd932d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b007387-6e38-46f4-b1f6-43d948fd932d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:16:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:07,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:07,636 [243] [DEBUG] [app] Starting request: urn:request:468dfb7c-9e71-480d-b509-33749f297ae7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:07,636 [243] [DEBUG] [app] Ending request: urn:request:468dfb7c-9e71-480d-b509-33749f297ae7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:468dfb7c-9e71-480d-b509-33749f297ae7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:07,636 [239] [DEBUG] [app] Starting request: urn:request:b9008da9-b455-45b6-9f65-b4c783d04148 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:07,637 [239] [DEBUG] [app] Ending request: urn:request:b9008da9-b455-45b6-9f65-b4c783d04148 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9008da9-b455-45b6-9f65-b4c783d04148', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:07,648 [239] [DEBUG] [app] Ending request: urn:request:5374d3c9-8ec9-42b7-ae3d-3afd8efcb905 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5374d3c9-8ec9-42b7-ae3d-3afd8efcb905', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:07,648 [241] [DEBUG] [app] Ending request: urn:request:a3084a9e-02f5-493e-8f2d-10861e5169ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a3084a9e-02f5-493e-8f2d-10861e5169ae', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:16:07,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 05:16:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:16:07,842 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:16:08,026 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:16:08,030 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:16:08,033 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:16:08,037 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:16:08,039 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:16:08,042 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:16:08,045 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:16:08,047 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:16:08,050 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:16:08,281 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:16:08,284 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:16:08,286 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:16:08,290 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:16:08,293 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:16:08,580 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:08,588 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:08,589 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:08,594 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:16:08,621 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:16:08,670 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:16:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:18.735066+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:16:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:18 UTC)" (scheduled at 2023-01-17 05:16:08.735066+00:00) notificationworker stdout | 2023-01-17 05:16:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 8, 735963), True, datetime.datetime(2023, 1, 17, 5, 16, 8, 735963), 0, 'notification/%', 50, 1, 0]) blobuploadcleanupworker stdout | 2023-01-17 05:16:08,741 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:16:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:18 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:08,780 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:16:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:22.837365+00:00 (in 13.606890 seconds) securityworker stdout | 2023-01-17 05:16:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:39 UTC)" (scheduled at 2023-01-17 05:16:09.230038+00:00) securityworker stdout | 2023-01-17 05:16:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:16:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:16:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:09,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:16:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:39 UTC)" executed successfully builder stdout | 2023-01-17 05:16:18,604 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:18,613 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:18,613 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:18,619 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:28.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:16:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:28 UTC)" (scheduled at 2023-01-17 05:16:18.735066+00:00) notificationworker stdout | 2023-01-17 05:16:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 18, 735956), True, datetime.datetime(2023, 1, 17, 5, 16, 18, 735956), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:16:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:25.334138+00:00 (in 5.091449 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:16:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:20 UTC)" (scheduled at 2023-01-17 05:16:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:16:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:16:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 20, 243088), True, datetime.datetime(2023, 1, 17, 5, 16, 20, 243088), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:16:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:16:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:16:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:16:22,625 [243] [DEBUG] [app] Starting request: urn:request:22447526-c7b4-4471-ac29-15bc1d8b2b05 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:22,625 [241] [DEBUG] [app] Starting request: urn:request:ef8a14de-41cc-4fd7-98ba-c199c0109ba5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:22,631 [240] [DEBUG] [app] Starting request: urn:request:e6cbc05d-a410-4370-b2a0-1936794ff1c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,631 [234] [DEBUG] [app] Starting request: urn:request:6751173e-cf7b-418b-934a-9d6fd01b8681 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,631 [240] [DEBUG] [app] Ending request: urn:request:e6cbc05d-a410-4370-b2a0-1936794ff1c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e6cbc05d-a410-4370-b2a0-1936794ff1c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,631 [234] [DEBUG] [app] Ending request: urn:request:6751173e-cf7b-418b-934a-9d6fd01b8681 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6751173e-cf7b-418b-934a-9d6fd01b8681', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:16:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:16:22,632 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:22,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:22,637 [239] [DEBUG] [app] Starting request: urn:request:162a83be-197f-4d66-b08f-7aa2f388bd95 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:22,637 [244] [DEBUG] [app] Starting request: urn:request:6a21f8e7-e037-43d9-903d-d87c92494702 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:22,638 [239] [DEBUG] [app] Ending request: urn:request:162a83be-197f-4d66-b08f-7aa2f388bd95 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:162a83be-197f-4d66-b08f-7aa2f388bd95', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:22,638 [244] [DEBUG] [app] Ending request: urn:request:6a21f8e7-e037-43d9-903d-d87c92494702 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a21f8e7-e037-43d9-903d-d87c92494702', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:22,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:22,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:22,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:22,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:22,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:22,639 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:22,644 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:22,644 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:22,649 [241] [DEBUG] [app] Ending request: urn:request:ef8a14de-41cc-4fd7-98ba-c199c0109ba5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ef8a14de-41cc-4fd7-98ba-c199c0109ba5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:22,649 [243] [DEBUG] [app] Ending request: urn:request:22447526-c7b4-4471-ac29-15bc1d8b2b05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:22447526-c7b4-4471-ac29-15bc1d8b2b05', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:22,650 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:22,650 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) gunicorn-web stdout | 2023-01-17 05:16:22,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:16:22,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:16:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:39.230038+00:00 (in 16.392268 seconds) securityworker stdout | 2023-01-17 05:16:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:52 UTC)" (scheduled at 2023-01-17 05:16:22.837365+00:00) securityworker stdout | 2023-01-17 05:16:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:22,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:16:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:16:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:44.641497+00:00 (in 19.999164 seconds) exportactionlogsworker stdout | 2023-01-17 05:16:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:24 UTC)" (scheduled at 2023-01-17 05:16:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:16:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:16:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:16:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:39.333860+00:00 (in 13.999224 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:16:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:25 UTC)" (scheduled at 2023-01-17 05:16:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:16:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:16:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:16:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:16:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:57.226424+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 05:16:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:57 UTC)" (scheduled at 2023-01-17 05:16:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:16:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 16, 27, 227275), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:16:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:16:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:16:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:57 UTC)" executed successfully builder stdout | 2023-01-17 05:16:28,629 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:28,640 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:28,640 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:28,645 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:38.735066+00:00 (in 9.999492 seconds) notificationworker stdout | 2023-01-17 05:16:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:38 UTC)" (scheduled at 2023-01-17 05:16:28.735066+00:00) notificationworker stdout | 2023-01-17 05:16:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 28, 736072), True, datetime.datetime(2023, 1, 17, 5, 16, 28, 736072), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:28,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:16:29,702 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:16:29,901 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:16:30,181 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:16:30,969 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:16:30,978 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:16:32,182 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:16:32,975 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:16:32,981 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:16:33,429 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:16:34,314 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:16:34,410 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:16:35,339 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:16:35,706 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:16:35,709 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:16:35,711 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:16:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:16:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:05.848675+00:00 (in 29.999488 seconds) gcworker stdout | 2023-01-17 05:16:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:05 UTC)" (scheduled at 2023-01-17 05:16:35.848675+00:00) gcworker stdout | 2023-01-17 05:16:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722995849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:16:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:16:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:16:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:16:36,069 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:16:36,174 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:16:37,319 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:16:37,626 [241] [DEBUG] [app] Starting request: urn:request:5f15c198-ba9d-4732-8a0d-962544d732b0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:37,626 [244] [DEBUG] [app] Starting request: urn:request:2c117290-12ad-4063-bcb6-89bfa97d66b6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:37,631 [232] [DEBUG] [app] Starting request: urn:request:27834b03-7ad1-4aa8-bc82-6027d5406fdd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,631 [240] [DEBUG] [app] Starting request: urn:request:02c72e63-370d-425b-ae7c-fc468129c827 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,631 [232] [DEBUG] [app] Ending request: urn:request:27834b03-7ad1-4aa8-bc82-6027d5406fdd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27834b03-7ad1-4aa8-bc82-6027d5406fdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:16:37,631 [240] [DEBUG] [app] Ending request: urn:request:02c72e63-370d-425b-ae7c-fc468129c827 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02c72e63-370d-425b-ae7c-fc468129c827', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:16:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:16:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:16:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:37,636 [243] [DEBUG] [app] Starting request: urn:request:c4ed4f89-471c-402e-b778-b7915920e026 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:37,636 [243] [DEBUG] [app] Ending request: urn:request:c4ed4f89-471c-402e-b778-b7915920e026 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4ed4f89-471c-402e-b778-b7915920e026', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:37,636 [239] [DEBUG] [app] Starting request: urn:request:a876d4b3-765a-4b29-9499-ac283d10d40c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:37,637 [239] [DEBUG] [app] Ending request: urn:request:a876d4b3-765a-4b29-9499-ac283d10d40c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a876d4b3-765a-4b29-9499-ac283d10d40c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:37,647 [241] [DEBUG] [app] Ending request: urn:request:5f15c198-ba9d-4732-8a0d-962544d732b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5f15c198-ba9d-4732-8a0d-962544d732b0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:37,647 [244] [DEBUG] [app] Ending request: urn:request:2c117290-12ad-4063-bcb6-89bfa97d66b6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2c117290-12ad-4063-bcb6-89bfa97d66b6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:16:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:16:37,877 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:16:38,034 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:16:38,038 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:16:38,041 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:16:38,043 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:16:38,047 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:16:38,049 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:16:38,052 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:16:38,055 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:16:38,057 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:16:38,288 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:16:38,290 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:16:38,292 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:16:38,295 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:16:38,299 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:16:38,637 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:16:38,656 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:38,664 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:38,664 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:38,670 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:16:38,690 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:16:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:43.935629+00:00 (in 5.200041 seconds) notificationworker stdout | 2023-01-17 05:16:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:48 UTC)" (scheduled at 2023-01-17 05:16:38.735066+00:00) notificationworker stdout | 2023-01-17 05:16:38,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 38, 736042), True, datetime.datetime(2023, 1, 17, 5, 16, 38, 736042), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:48 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:16:38,748 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:16:38,802 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} repositorygcworker stdout | 2023-01-17 05:16:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:16:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:48.738458+00:00 (in 9.707229 seconds) repositorygcworker stdout | 2023-01-17 05:16:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:39 UTC)" (scheduled at 2023-01-17 05:16:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:16:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:16:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 39, 31622), True, datetime.datetime(2023, 1, 17, 5, 16, 39, 31622), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:16:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:16:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:16:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:16:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:52.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 05:16:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:09 UTC)" (scheduled at 2023-01-17 05:16:39.230038+00:00) securityworker stdout | 2023-01-17 05:16:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:16:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:16:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:39,244 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:39,245 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:16:39,245 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:16:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:20.242198+00:00 (in 40.907856 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:16:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:39 UTC)" (scheduled at 2023-01-17 05:16:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:16:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 16, 39, 334726), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:16:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 39, 334726), True, datetime.datetime(2023, 1, 17, 5, 16, 39, 334726), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:16:39,344 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 39, 334726), True, datetime.datetime(2023, 1, 17, 5, 16, 39, 334726), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 16, 39, 334726), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:16:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:16:39,346 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:48.735066+00:00 (in 4.798813 seconds) notificationworker stdout | 2023-01-17 05:16:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:43 UTC)" (scheduled at 2023-01-17 05:16:43.935629+00:00) notificationworker stdout | 2023-01-17 05:16:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:16:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:16:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:16:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:48.830274+00:00 (in 4.188252 seconds) exportactionlogsworker stdout | 2023-01-17 05:16:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:44 UTC)" (scheduled at 2023-01-17 05:16:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:16:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 16, 44, 642419), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:16:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 44, 642419), True, datetime.datetime(2023, 1, 17, 5, 16, 44, 642419), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:16:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 44, 642419), True, datetime.datetime(2023, 1, 17, 5, 16, 44, 642419), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 16, 44, 642419), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:16:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:16:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:16:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:16:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:59.337413+00:00 (in 11.404479 seconds) namespacegcworker stdout | 2023-01-17 05:16:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:47 UTC)" (scheduled at 2023-01-17 05:16:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:16:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:16:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:47 UTC)" executed successfully builder stdout | 2023-01-17 05:16:48,671 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:48,679 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:48,679 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:48,685 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:58.735066+00:00 (in 9.999453 seconds) notificationworker stdout | 2023-01-17 05:16:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:58 UTC)" (scheduled at 2023-01-17 05:16:48.735066+00:00) notificationworker stdout | 2023-01-17 05:16:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 48, 736072), True, datetime.datetime(2023, 1, 17, 5, 16, 48, 736072), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:16:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:16:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:04.738062+00:00 (in 15.999144 seconds) repositorygcworker stdout | 2023-01-17 05:16:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:48 UTC)" (scheduled at 2023-01-17 05:16:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:16:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:16:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:16:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:16:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:24.641816+00:00 (in 35.811083 seconds) exportactionlogsworker stdout | 2023-01-17 05:16:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:48 UTC)" (scheduled at 2023-01-17 05:16:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:16:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:16:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 48, 831100), True, datetime.datetime(2023, 1, 17, 5, 16, 48, 831100), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:16:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:16:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:16:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:16:52,625 [239] [DEBUG] [app] Starting request: urn:request:3fa375ab-ef21-4f90-8bfd-2067e47d614c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:52,625 [244] [DEBUG] [app] Starting request: urn:request:8484e4de-e422-4fa5-a94f-8956a30183c4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:16:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:52,630 [232] [DEBUG] [app] Starting request: urn:request:3b7bef15-e060-45ac-83bc-48ba0c93e963 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:52,631 [240] [DEBUG] [app] Starting request: urn:request:d011f3d9-156e-4b76-9de2-2b3373cadbcb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:52,631 [232] [DEBUG] [app] Ending request: urn:request:3b7bef15-e060-45ac-83bc-48ba0c93e963 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b7bef15-e060-45ac-83bc-48ba0c93e963', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:16:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:16:52,631 [240] [DEBUG] [app] Ending request: urn:request:d011f3d9-156e-4b76-9de2-2b3373cadbcb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d011f3d9-156e-4b76-9de2-2b3373cadbcb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:16:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:16:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:52,636 [244] [DEBUG] [app] Starting request: urn:request:86a37a36-c73a-4b73-b5d3-ae085dac02d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:52,636 [244] [DEBUG] [app] Ending request: urn:request:86a37a36-c73a-4b73-b5d3-ae085dac02d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86a37a36-c73a-4b73-b5d3-ae085dac02d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:52,636 [243] [DEBUG] [app] Starting request: urn:request:6b4d710d-0574-465f-8126-8d1bb7e0d4df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:52,637 [243] [DEBUG] [app] Ending request: urn:request:6b4d710d-0574-465f-8126-8d1bb7e0d4df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b4d710d-0574-465f-8126-8d1bb7e0d4df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:52,647 [239] [DEBUG] [app] Ending request: urn:request:3fa375ab-ef21-4f90-8bfd-2067e47d614c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3fa375ab-ef21-4f90-8bfd-2067e47d614c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:52,648 [244] [DEBUG] [app] Ending request: urn:request:8484e4de-e422-4fa5-a94f-8956a30183c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8484e4de-e422-4fa5-a94f-8956a30183c4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:16:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:16:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:09.230038+00:00 (in 16.392234 seconds) securityworker stdout | 2023-01-17 05:16:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:22 UTC)" (scheduled at 2023-01-17 05:16:52.837365+00:00) securityworker stdout | 2023-01-17 05:16:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:16:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:16:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:27.226424+00:00 (in 29.999545 seconds) buildlogsarchiver stdout | 2023-01-17 05:16:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:27 UTC)" (scheduled at 2023-01-17 05:16:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:16:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 16, 57, 227216), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:16:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:16:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:16:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:27 UTC)" executed successfully builder stdout | 2023-01-17 05:16:58,695 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:58,704 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:58,704 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:58,710 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:08.735066+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:16:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:08 UTC)" (scheduled at 2023-01-17 05:16:58.735066+00:00) notificationworker stdout | 2023-01-17 05:16:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 58, 736065), True, datetime.datetime(2023, 1, 17, 5, 16, 58, 736065), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:58,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:16:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:16:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:47.932490+00:00 (in 48.594599 seconds) namespacegcworker stdout | 2023-01-17 05:16:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:59 UTC)" (scheduled at 2023-01-17 05:16:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:16:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:16:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 59, 338338), True, datetime.datetime(2023, 1, 17, 5, 16, 59, 338338), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:16:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:16:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:16:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:16:59,728 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:16:59,937 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:17:00,212 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:17:00,989 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:17:00,995 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:17:02,217 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:17:02,991 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:17:03,002 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:17:03,450 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:17:04,350 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:17:04,429 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} repositorygcworker stdout | 2023-01-17 05:17:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:17:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:39.030814+00:00 (in 34.292297 seconds) repositorygcworker stdout | 2023-01-17 05:17:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:22:04 UTC)" (scheduled at 2023-01-17 05:17:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:17:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 17, 4, 738915), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:17:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 17, 4, 738915), True, datetime.datetime(2023, 1, 17, 5, 17, 4, 738915), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:17:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 17, 4, 738915), True, datetime.datetime(2023, 1, 17, 5, 17, 4, 738915), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 17, 4, 738915), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:17:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:17:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:22:04 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:17:05,354 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:17:05,712 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:17:05,715 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:17:05,717 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:17:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:17:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:35.848675+00:00 (in 29.999507 seconds) gcworker stdout | 2023-01-17 05:17:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:35 UTC)" (scheduled at 2023-01-17 05:17:05.848675+00:00) gcworker stdout | 2023-01-17 05:17:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723025849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:17:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:17:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:17:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:17:06,102 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:17:06,208 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:17:07,349 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:17:07,625 [243] [DEBUG] [app] Starting request: urn:request:8e333340-4f44-462b-9c7f-eff168ab17e5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:07,625 [244] [DEBUG] [app] Starting request: urn:request:d2cd595c-413c-430f-a182-dec7ff74778d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:07,629 [232] [DEBUG] [app] Starting request: urn:request:009b9228-1152-4904-8313-2d24b73e72a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:07,630 [242] [DEBUG] [app] Starting request: urn:request:31c49b90-34b2-4b97-9012-62c1433d0e19 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:07,630 [232] [DEBUG] [app] Ending request: urn:request:009b9228-1152-4904-8313-2d24b73e72a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:009b9228-1152-4904-8313-2d24b73e72a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:17:07,630 [242] [DEBUG] [app] Ending request: urn:request:31c49b90-34b2-4b97-9012-62c1433d0e19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31c49b90-34b2-4b97-9012-62c1433d0e19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:17:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:07,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:17:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:07,635 [239] [DEBUG] [app] Starting request: urn:request:b225bddc-2a71-4b96-97e8-150b8941847c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:07,635 [239] [DEBUG] [app] Ending request: urn:request:b225bddc-2a71-4b96-97e8-150b8941847c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b225bddc-2a71-4b96-97e8-150b8941847c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:07,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:07,635 [241] [DEBUG] [app] Starting request: urn:request:71f22d5a-a69d-4144-a5bf-f79caa52a8ab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:07,636 [241] [DEBUG] [app] Ending request: urn:request:71f22d5a-a69d-4144-a5bf-f79caa52a8ab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71f22d5a-a69d-4144-a5bf-f79caa52a8ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:07,646 [244] [DEBUG] [app] Ending request: urn:request:d2cd595c-413c-430f-a182-dec7ff74778d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d2cd595c-413c-430f-a182-dec7ff74778d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:07,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:17:07,647 [243] [DEBUG] [app] Ending request: urn:request:8e333340-4f44-462b-9c7f-eff168ab17e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e333340-4f44-462b-9c7f-eff168ab17e5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) notificationworker stdout | 2023-01-17 05:17:07,913 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:17:08,042 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:17:08,046 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:17:08,048 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:17:08,051 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:17:08,056 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:17:08,058 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:17:08,060 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:17:08,063 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:17:08,065 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:17:08,295 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:17:08,298 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:17:08,301 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:17:08,303 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:17:08,305 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:17:08,673 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:17:08,717 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue expiredappspecifictokenworker stdout | 2023-01-17 05:17:08,721 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} builder stdout | 2023-01-17 05:17:08,726 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:08,726 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:08,731 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:18.735066+00:00 (in 9.999381 seconds) notificationworker stdout | 2023-01-17 05:17:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:18 UTC)" (scheduled at 2023-01-17 05:17:08.735066+00:00) notificationworker stdout | 2023-01-17 05:17:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 8, 736172), True, datetime.datetime(2023, 1, 17, 5, 17, 8, 736172), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:18 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:17:08,769 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:17:08,836 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:17:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:22.837365+00:00 (in 13.606889 seconds) securityworker stdout | 2023-01-17 05:17:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:39 UTC)" (scheduled at 2023-01-17 05:17:09.230038+00:00) securityworker stdout | 2023-01-17 05:17:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:17:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:17:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:17:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:17:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:17:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:17:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:28.735066+00:00 (in 9.999373 seconds) notificationworker stdout | 2023-01-17 05:17:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:28 UTC)" (scheduled at 2023-01-17 05:17:18.735066+00:00) notificationworker stdout | 2023-01-17 05:17:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 18, 736119), True, datetime.datetime(2023, 1, 17, 5, 17, 18, 736119), 0, 'notification/%', 50, 1, 0]) builder stdout | 2023-01-17 05:17:18,741 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue notificationworker stdout | 2023-01-17 05:17:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:28 UTC)" executed successfully builder stdout | 2023-01-17 05:17:18,748 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:18,749 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:18,754 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:17:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:17:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:25.334138+00:00 (in 5.091436 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:17:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:20 UTC)" (scheduled at 2023-01-17 05:17:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:17:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:17:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 20, 243123), True, datetime.datetime(2023, 1, 17, 5, 17, 20, 243123), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:17:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:17:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:17:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:17:22,625 [239] [DEBUG] [app] Starting request: urn:request:e6d43726-af9e-4f21-8f44-d6b651d1223e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:22,625 [244] [DEBUG] [app] Starting request: urn:request:d609b236-02fb-4b68-9dd3-07b3c38f934a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:22,630 [236] [DEBUG] [app] Starting request: urn:request:6207ac42-8a0f-4844-b507-0fc9db67ca57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,631 [242] [DEBUG] [app] Starting request: urn:request:c4edb4e3-4ef0-4601-92be-530e90ce0372 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,631 [236] [DEBUG] [app] Ending request: urn:request:6207ac42-8a0f-4844-b507-0fc9db67ca57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6207ac42-8a0f-4844-b507-0fc9db67ca57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:17:22,631 [242] [DEBUG] [app] Ending request: urn:request:c4edb4e3-4ef0-4601-92be-530e90ce0372 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c4edb4e3-4ef0-4601-92be-530e90ce0372', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:17:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:17:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:22,636 [243] [DEBUG] [app] Starting request: urn:request:97ae326b-77df-4ba7-a14e-c7f758d0555f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:22,636 [244] [DEBUG] [app] Starting request: urn:request:776eea93-bb59-452d-8cef-0a00bc4e71e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:22,637 [243] [DEBUG] [app] Ending request: urn:request:97ae326b-77df-4ba7-a14e-c7f758d0555f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97ae326b-77df-4ba7-a14e-c7f758d0555f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:22,637 [244] [DEBUG] [app] Ending request: urn:request:776eea93-bb59-452d-8cef-0a00bc4e71e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:776eea93-bb59-452d-8cef-0a00bc4e71e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:22,648 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:22,650 [239] [DEBUG] [app] Ending request: urn:request:e6d43726-af9e-4f21-8f44-d6b651d1223e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6d43726-af9e-4f21-8f44-d6b651d1223e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:22,650 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:22,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) gunicorn-web stdout | 2023-01-17 05:17:22,653 [244] [DEBUG] [app] Ending request: urn:request:d609b236-02fb-4b68-9dd3-07b3c38f934a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d609b236-02fb-4b68-9dd3-07b3c38f934a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:22,653 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:22,654 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityworker stdout | 2023-01-17 05:17:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:39.230038+00:00 (in 16.392265 seconds) securityworker stdout | 2023-01-17 05:17:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:52 UTC)" (scheduled at 2023-01-17 05:17:22.837365+00:00) securityworker stdout | 2023-01-17 05:17:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:17:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:17:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:48.830274+00:00 (in 24.187983 seconds) exportactionlogsworker stdout | 2023-01-17 05:17:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:24 UTC)" (scheduled at 2023-01-17 05:17:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:17:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:17:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:17:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:17:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:20.242198+00:00 (in 54.907567 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:17:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:25 UTC)" (scheduled at 2023-01-17 05:17:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:17:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:17:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:17:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:17:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:57.226424+00:00 (in 29.999559 seconds) buildlogsarchiver stdout | 2023-01-17 05:17:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:57 UTC)" (scheduled at 2023-01-17 05:17:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:17:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 17, 27, 227241), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:17:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:17:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:17:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:38.735066+00:00 (in 9.999492 seconds) notificationworker stdout | 2023-01-17 05:17:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:38 UTC)" (scheduled at 2023-01-17 05:17:28.735066+00:00) notificationworker stdout | 2023-01-17 05:17:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 28, 736078), True, datetime.datetime(2023, 1, 17, 5, 17, 28, 736078), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:38 UTC)" executed successfully builder stdout | 2023-01-17 05:17:28,758 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:28,765 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:28,766 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:28,771 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:29,758 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:17:29,971 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:17:30,225 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:17:31,015 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:17:31,024 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:17:32,253 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:17:33,025 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:17:33,028 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:17:33,470 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:17:34,378 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:17:34,465 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:17:35,388 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:17:35,720 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:17:35,723 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:17:35,726 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:17:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:17:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:05.848675+00:00 (in 29.999553 seconds) gcworker stdout | 2023-01-17 05:17:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:05 UTC)" (scheduled at 2023-01-17 05:17:35.848675+00:00) gcworker stdout | 2023-01-17 05:17:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723055849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:17:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:17:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:17:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:17:36,137 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:17:36,216 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:17:37,359 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:17:37,625 [239] [DEBUG] [app] Starting request: urn:request:f4b478c1-eaa6-4804-a474-b97e6213fda3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:37,625 [243] [DEBUG] [app] Starting request: urn:request:a1fdbc1e-9d86-4ace-aeac-b017ce4614f1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:37,630 [237] [DEBUG] [app] Starting request: urn:request:b714aa3d-0fc1-447e-a861-7309c2c46400 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:37,631 [242] [DEBUG] [app] Starting request: urn:request:b680ec65-e995-42d8-a828-a16184c0f6d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:37,631 [237] [DEBUG] [app] Ending request: urn:request:b714aa3d-0fc1-447e-a861-7309c2c46400 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b714aa3d-0fc1-447e-a861-7309c2c46400', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:17:37,631 [242] [DEBUG] [app] Ending request: urn:request:b680ec65-e995-42d8-a828-a16184c0f6d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b680ec65-e995-42d8-a828-a16184c0f6d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:17:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:17:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:37,635 [239] [DEBUG] [app] Starting request: urn:request:fdfb89f9-7363-4cfe-8eaf-3f4f3619e980 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:37,635 [239] [DEBUG] [app] Ending request: urn:request:fdfb89f9-7363-4cfe-8eaf-3f4f3619e980 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdfb89f9-7363-4cfe-8eaf-3f4f3619e980', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:37,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:37,637 [244] [DEBUG] [app] Starting request: urn:request:76cc863f-25af-4e99-b346-be3eb4fe39eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:37,637 [244] [DEBUG] [app] Ending request: urn:request:76cc863f-25af-4e99-b346-be3eb4fe39eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76cc863f-25af-4e99-b346-be3eb4fe39eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:37,646 [243] [DEBUG] [app] Ending request: urn:request:a1fdbc1e-9d86-4ace-aeac-b017ce4614f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a1fdbc1e-9d86-4ace-aeac-b017ce4614f1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:37,646 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:17:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:17:37,648 [239] [DEBUG] [app] Ending request: urn:request:f4b478c1-eaa6-4804-a474-b97e6213fda3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4b478c1-eaa6-4804-a474-b97e6213fda3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) notificationworker stdout | 2023-01-17 05:17:37,948 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:17:38,049 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:17:38,054 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:17:38,057 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:17:38,059 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:17:38,063 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:17:38,067 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:17:38,069 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:17:38,072 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:17:38,074 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:17:38,302 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:17:38,304 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:17:38,307 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:17:38,310 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:17:38,313 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:17:38,708 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:17:38,734 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:17:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:43.935629+00:00 (in 5.200087 seconds) notificationworker stdout | 2023-01-17 05:17:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:48 UTC)" (scheduled at 2023-01-17 05:17:38.735066+00:00) notificationworker stdout | 2023-01-17 05:17:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 38, 735939), True, datetime.datetime(2023, 1, 17, 5, 17, 38, 735939), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:48 UTC)" executed successfully builder stdout | 2023-01-17 05:17:38,781 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue blobuploadcleanupworker stdout | 2023-01-17 05:17:38,787 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:17:38,789 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:38,789 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:38,795 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:17:38,845 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} repositorygcworker stdout | 2023-01-17 05:17:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:17:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:48.738458+00:00 (in 9.707180 seconds) repositorygcworker stdout | 2023-01-17 05:17:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:39 UTC)" (scheduled at 2023-01-17 05:17:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:17:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:17:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 39, 31702), True, datetime.datetime(2023, 1, 17, 5, 17, 39, 31702), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:17:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:17:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:17:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:17:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:52.837365+00:00 (in 13.606885 seconds) securityworker stdout | 2023-01-17 05:17:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:09 UTC)" (scheduled at 2023-01-17 05:17:39.230038+00:00) securityworker stdout | 2023-01-17 05:17:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:17:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:17:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:39,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:17:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:48.735066+00:00 (in 4.798986 seconds) notificationworker stdout | 2023-01-17 05:17:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:43 UTC)" (scheduled at 2023-01-17 05:17:43.935629+00:00) notificationworker stdout | 2023-01-17 05:17:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:17:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:17:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:17:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:59.337413+00:00 (in 11.404440 seconds) namespacegcworker stdout | 2023-01-17 05:17:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:47 UTC)" (scheduled at 2023-01-17 05:17:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:17:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:17:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:58.735066+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 05:17:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:58 UTC)" (scheduled at 2023-01-17 05:17:48.735066+00:00) notificationworker stdout | 2023-01-17 05:17:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:48,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 48, 736007), True, datetime.datetime(2023, 1, 17, 5, 17, 48, 736007), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:17:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:17:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:39.030814+00:00 (in 50.291921 seconds) repositorygcworker stdout | 2023-01-17 05:17:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:48 UTC)" (scheduled at 2023-01-17 05:17:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:17:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:17:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:58 UTC)" executed successfully builder stdout | 2023-01-17 05:17:48,796 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:48,804 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:48,804 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:48,810 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:17:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:17:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:24.641816+00:00 (in 35.811081 seconds) exportactionlogsworker stdout | 2023-01-17 05:17:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:48 UTC)" (scheduled at 2023-01-17 05:17:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:17:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:17:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 48, 831124), True, datetime.datetime(2023, 1, 17, 5, 17, 48, 831124), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:17:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:17:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:17:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:17:52,625 [241] [DEBUG] [app] Starting request: urn:request:ef52ff7a-1845-479c-912e-850a3c9af143 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:52,625 [244] [DEBUG] [app] Starting request: urn:request:62c53f90-ba11-4523-b532-0b7dfb119930 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:17:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:52,630 [242] [DEBUG] [app] Starting request: urn:request:58652e37-5692-466e-8dd4-e02a0077ea5d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:52,630 [236] [DEBUG] [app] Starting request: urn:request:01be279e-0386-4759-9931-946c2a71e50a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:52,631 [242] [DEBUG] [app] Ending request: urn:request:58652e37-5692-466e-8dd4-e02a0077ea5d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58652e37-5692-466e-8dd4-e02a0077ea5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:17:52,631 [236] [DEBUG] [app] Ending request: urn:request:01be279e-0386-4759-9931-946c2a71e50a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01be279e-0386-4759-9931-946c2a71e50a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:17:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:17:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:17:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:52,635 [239] [DEBUG] [app] Starting request: urn:request:15ac19c1-2a48-4c31-a3b6-b06b8f95d57b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:52,636 [239] [DEBUG] [app] Ending request: urn:request:15ac19c1-2a48-4c31-a3b6-b06b8f95d57b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15ac19c1-2a48-4c31-a3b6-b06b8f95d57b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:52,636 [243] [DEBUG] [app] Starting request: urn:request:887d48f3-42a8-4a19-8dcf-6f5f699d0bec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:52,636 [243] [DEBUG] [app] Ending request: urn:request:887d48f3-42a8-4a19-8dcf-6f5f699d0bec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:887d48f3-42a8-4a19-8dcf-6f5f699d0bec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:52,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:52,647 [244] [DEBUG] [app] Ending request: urn:request:62c53f90-ba11-4523-b532-0b7dfb119930 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62c53f90-ba11-4523-b532-0b7dfb119930', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:52,647 [241] [DEBUG] [app] Ending request: urn:request:ef52ff7a-1845-479c-912e-850a3c9af143 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ef52ff7a-1845-479c-912e-850a3c9af143', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:52,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:17:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:17:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:09.230038+00:00 (in 16.392258 seconds) securityworker stdout | 2023-01-17 05:17:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:22 UTC)" (scheduled at 2023-01-17 05:17:52.837365+00:00) securityworker stdout | 2023-01-17 05:17:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:17:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:17:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:27.226424+00:00 (in 29.999548 seconds) buildlogsarchiver stdout | 2023-01-17 05:17:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:27 UTC)" (scheduled at 2023-01-17 05:17:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:17:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 17, 57, 227228), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:17:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:17:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:17:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:08.735066+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:17:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:08 UTC)" (scheduled at 2023-01-17 05:17:58.735066+00:00) notificationworker stdout | 2023-01-17 05:17:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 58, 736035), True, datetime.datetime(2023, 1, 17, 5, 17, 58, 736035), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:58,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:08 UTC)" executed successfully builder stdout | 2023-01-17 05:17:58,817 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:58,825 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:58,825 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:58,830 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:17:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:17:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:47.932490+00:00 (in 48.594631 seconds) namespacegcworker stdout | 2023-01-17 05:17:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:59 UTC)" (scheduled at 2023-01-17 05:17:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:17:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:17:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 59, 338270), True, datetime.datetime(2023, 1, 17, 5, 17, 59, 338270), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:17:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:17:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:17:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:17:59,781 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:18:00,001 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:18:00,254 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:18:01,039 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:18:01,049 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:18:02,289 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:18:03,057 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:18:03,060 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:18:03,476 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:18:04,385 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:18:04,485 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:18:05,424 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:18:05,727 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:18:05,729 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:18:05,732 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:18:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:18:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:35.848675+00:00 (in 29.999323 seconds) gcworker stdout | 2023-01-17 05:18:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:35 UTC)" (scheduled at 2023-01-17 05:18:05.848675+00:00) gcworker stdout | 2023-01-17 05:18:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723085849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:18:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:18:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:18:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:18:06,173 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:18:06,223 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:18:07,395 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:18:07,625 [239] [DEBUG] [app] Starting request: urn:request:0cca61da-9151-4055-a136-1e0273e5b1b6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:07,625 [244] [DEBUG] [app] Starting request: urn:request:7184955b-af51-4009-912c-64c2c29c6866 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:07,630 [232] [DEBUG] [app] Starting request: urn:request:d9a3fb90-750f-4602-8f8c-da0af5e884ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:07,631 [240] [DEBUG] [app] Starting request: urn:request:3f2896ce-6331-45ef-8cee-075f1364acdd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:07,631 [232] [DEBUG] [app] Ending request: urn:request:d9a3fb90-750f-4602-8f8c-da0af5e884ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d9a3fb90-750f-4602-8f8c-da0af5e884ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:07,631 [240] [DEBUG] [app] Ending request: urn:request:3f2896ce-6331-45ef-8cee-075f1364acdd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f2896ce-6331-45ef-8cee-075f1364acdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:18:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:18:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:07,636 [241] [DEBUG] [app] Starting request: urn:request:6572ec6a-9ab7-4ecb-8d95-ab506db13e49 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:07,636 [239] [DEBUG] [app] Starting request: urn:request:bade47fa-720f-43f5-b6a1-07c4bd66c502 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:07,637 [241] [DEBUG] [app] Ending request: urn:request:6572ec6a-9ab7-4ecb-8d95-ab506db13e49 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6572ec6a-9ab7-4ecb-8d95-ab506db13e49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:07,637 [239] [DEBUG] [app] Ending request: urn:request:bade47fa-720f-43f5-b6a1-07c4bd66c502 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bade47fa-720f-43f5-b6a1-07c4bd66c502', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:18:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:07,648 [244] [DEBUG] [app] Ending request: urn:request:7184955b-af51-4009-912c-64c2c29c6866 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7184955b-af51-4009-912c-64c2c29c6866', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:07,648 [239] [DEBUG] [app] Ending request: urn:request:0cca61da-9151-4055-a136-1e0273e5b1b6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0cca61da-9151-4055-a136-1e0273e5b1b6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:18:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:18:07,959 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:18:08,058 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:18:08,062 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:18:08,064 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:18:08,068 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:18:08,070 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:18:08,074 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:18:08,078 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:18:08,081 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:18:08,083 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:18:08,308 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:18:08,312 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:18:08,314 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:18:08,316 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:18:08,319 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gcworker stdout | 2023-01-17 05:18:08,725 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:18:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:18.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 05:18:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:18 UTC)" (scheduled at 2023-01-17 05:18:08.735066+00:00) notificationworker stdout | 2023-01-17 05:18:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 8, 735980), True, datetime.datetime(2023, 1, 17, 5, 18, 8, 735980), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:18 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:18:08,768 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:18:08,822 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:18:08,841 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:08,849 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:08,849 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:08,855 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:18:08,863 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:18:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:22.837365+00:00 (in 13.606845 seconds) securityworker stdout | 2023-01-17 05:18:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:39 UTC)" (scheduled at 2023-01-17 05:18:09.230038+00:00) securityworker stdout | 2023-01-17 05:18:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:18:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:18:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:18:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:28.735066+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:18:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:28 UTC)" (scheduled at 2023-01-17 05:18:18.735066+00:00) notificationworker stdout | 2023-01-17 05:18:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 18, 735943), True, datetime.datetime(2023, 1, 17, 5, 18, 18, 735943), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:28 UTC)" executed successfully builder stdout | 2023-01-17 05:18:18,856 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:18,864 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:18,864 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:18,870 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:18:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:18:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:25.334138+00:00 (in 5.091518 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:18:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:20 UTC)" (scheduled at 2023-01-17 05:18:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:18:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:18:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 20, 243094), True, datetime.datetime(2023, 1, 17, 5, 18, 20, 243094), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:18:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:18:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:18:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:18:22,625 [244] [DEBUG] [app] Starting request: urn:request:9a97a9ff-4d30-4261-b5c4-d40992c17e36 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:22,625 [243] [DEBUG] [app] Starting request: urn:request:f54f1a12-ced0-483c-80ab-de6e34012dbe (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:22,631 [237] [DEBUG] [app] Starting request: urn:request:822e3737-0912-4a16-afe4-b5e565dbb70d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:22,631 [240] [DEBUG] [app] Starting request: urn:request:2adb7fa7-1349-4388-9a60-9afb5cc2681e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:22,631 [237] [DEBUG] [app] Ending request: urn:request:822e3737-0912-4a16-afe4-b5e565dbb70d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:822e3737-0912-4a16-afe4-b5e565dbb70d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:18:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:18:22,631 [240] [DEBUG] [app] Ending request: urn:request:2adb7fa7-1349-4388-9a60-9afb5cc2681e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2adb7fa7-1349-4388-9a60-9afb5cc2681e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:18:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:18:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:22,636 [239] [DEBUG] [app] Starting request: urn:request:c72f74d3-11d7-4015-9350-fa0356313eac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:22,636 [239] [DEBUG] [app] Ending request: urn:request:c72f74d3-11d7-4015-9350-fa0356313eac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c72f74d3-11d7-4015-9350-fa0356313eac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:22,636 [243] [DEBUG] [app] Starting request: urn:request:3ee7f194-013e-439a-969c-ece8b8c776f9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:22,637 [243] [DEBUG] [app] Ending request: urn:request:3ee7f194-013e-439a-969c-ece8b8c776f9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ee7f194-013e-439a-969c-ece8b8c776f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:22,648 [244] [DEBUG] [app] Ending request: urn:request:9a97a9ff-4d30-4261-b5c4-d40992c17e36 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a97a9ff-4d30-4261-b5c4-d40992c17e36', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:22,648 [243] [DEBUG] [app] Ending request: urn:request:f54f1a12-ced0-483c-80ab-de6e34012dbe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f54f1a12-ced0-483c-80ab-de6e34012dbe', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:18:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:18:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:39.230038+00:00 (in 16.392197 seconds) securityworker stdout | 2023-01-17 05:18:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:52 UTC)" (scheduled at 2023-01-17 05:18:22.837365+00:00) securityworker stdout | 2023-01-17 05:18:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:18:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:18:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:48.830274+00:00 (in 24.187981 seconds) exportactionlogsworker stdout | 2023-01-17 05:18:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:24 UTC)" (scheduled at 2023-01-17 05:18:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:18:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:18:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:18:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:18:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:20.242198+00:00 (in 54.907567 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:18:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:25 UTC)" (scheduled at 2023-01-17 05:18:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:18:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:18:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:18:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:18:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:57.226424+00:00 (in 29.999573 seconds) buildlogsarchiver stdout | 2023-01-17 05:18:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:57 UTC)" (scheduled at 2023-01-17 05:18:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:18:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 18, 27, 227208), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:18:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:18:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:18:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:38.735066+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:18:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:38 UTC)" (scheduled at 2023-01-17 05:18:28.735066+00:00) notificationworker stdout | 2023-01-17 05:18:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 28, 735953), True, datetime.datetime(2023, 1, 17, 5, 18, 28, 735953), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:38 UTC)" executed successfully builder stdout | 2023-01-17 05:18:28,880 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:28,888 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:28,888 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:28,894 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:29,795 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:18:30,022 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:18:30,289 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:18:31,075 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:18:31,077 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:18:32,325 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:18:33,084 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:18:33,094 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:18:33,511 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:18:34,421 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:18:34,512 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:18:35,460 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:18:35,734 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:18:35,737 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:18:35,739 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:18:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:18:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:05.848675+00:00 (in 29.999484 seconds) gcworker stdout | 2023-01-17 05:18:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:05 UTC)" (scheduled at 2023-01-17 05:18:35.848675+00:00) gcworker stdout | 2023-01-17 05:18:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723115849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:18:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:18:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:18:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:18:36,208 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:18:36,256 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:18:37,415 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:18:37,626 [244] [DEBUG] [app] Starting request: urn:request:7e9965b6-f87c-43a5-ae4c-756967368dee (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:37,626 [243] [DEBUG] [app] Starting request: urn:request:7d18a5ca-d013-4e97-be31-895709ee7d4c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:37,631 [232] [DEBUG] [app] Starting request: urn:request:1db16aae-7001-4c81-a3e7-0b6190a679a9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,631 [235] [DEBUG] [app] Starting request: urn:request:0befaf19-17c3-458a-80fc-d7e5db094da6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,631 [232] [DEBUG] [app] Ending request: urn:request:1db16aae-7001-4c81-a3e7-0b6190a679a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1db16aae-7001-4c81-a3e7-0b6190a679a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,632 [235] [DEBUG] [app] Ending request: urn:request:0befaf19-17c3-458a-80fc-d7e5db094da6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0befaf19-17c3-458a-80fc-d7e5db094da6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:18:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:18:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:37,637 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:37,637 [244] [DEBUG] [app] Starting request: urn:request:1577a4e1-8bf5-49f0-8374-60a7768d7200 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:37,638 [239] [DEBUG] [app] Starting request: urn:request:01e256d6-6a52-4996-93d1-165f56678bfa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:37,638 [244] [DEBUG] [app] Ending request: urn:request:1577a4e1-8bf5-49f0-8374-60a7768d7200 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1577a4e1-8bf5-49f0-8374-60a7768d7200', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:37,638 [239] [DEBUG] [app] Ending request: urn:request:01e256d6-6a52-4996-93d1-165f56678bfa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01e256d6-6a52-4996-93d1-165f56678bfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:18:37,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:37,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:37,639 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:37,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:37,639 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:37,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:37,644 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:37,650 [243] [DEBUG] [app] Ending request: urn:request:7d18a5ca-d013-4e97-be31-895709ee7d4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d18a5ca-d013-4e97-be31-895709ee7d4c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:37,650 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:37,650 [244] [DEBUG] [app] Ending request: urn:request:7e9965b6-f87c-43a5-ae4c-756967368dee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e9965b6-f87c-43a5-ae4c-756967368dee', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:37,650 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) gunicorn-web stdout | 2023-01-17 05:18:37,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:18:37,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:18:37,989 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:18:38,067 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:18:38,069 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:18:38,071 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:18:38,074 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:18:38,077 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:18:38,080 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:18:38,086 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:18:38,089 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:18:38,091 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:18:38,314 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:18:38,318 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:18:38,320 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:18:38,323 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:18:38,326 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:18:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:43.935629+00:00 (in 5.200116 seconds) notificationworker stdout | 2023-01-17 05:18:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:48 UTC)" (scheduled at 2023-01-17 05:18:38.735066+00:00) notificationworker stdout | 2023-01-17 05:18:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 38, 735914), True, datetime.datetime(2023, 1, 17, 5, 18, 38, 735914), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:48 UTC)" executed successfully gcworker stdout | 2023-01-17 05:18:38,751 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:18:38,803 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:18:38,857 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:18:38,895 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue securityscanningnotificationworker stdout | 2023-01-17 05:18:38,898 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:18:38,903 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:38,903 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:38,909 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:18:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:18:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:48.738458+00:00 (in 9.707189 seconds) repositorygcworker stdout | 2023-01-17 05:18:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:39 UTC)" (scheduled at 2023-01-17 05:18:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:18:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:18:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 39, 31724), True, datetime.datetime(2023, 1, 17, 5, 18, 39, 31724), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:18:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:18:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:18:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:18:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:52.837365+00:00 (in 13.606892 seconds) securityworker stdout | 2023-01-17 05:18:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:09 UTC)" (scheduled at 2023-01-17 05:18:39.230038+00:00) securityworker stdout | 2023-01-17 05:18:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:18:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:18:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:18:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:48.735066+00:00 (in 4.798975 seconds) notificationworker stdout | 2023-01-17 05:18:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:43 UTC)" (scheduled at 2023-01-17 05:18:43.935629+00:00) notificationworker stdout | 2023-01-17 05:18:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:18:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:18:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:18:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:59.337413+00:00 (in 11.404487 seconds) namespacegcworker stdout | 2023-01-17 05:18:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:47 UTC)" (scheduled at 2023-01-17 05:18:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:18:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:18:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:56.935055+00:00 (in 8.199415 seconds) notificationworker stdout | 2023-01-17 05:18:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:58 UTC)" (scheduled at 2023-01-17 05:18:48.735066+00:00) notificationworker stdout | 2023-01-17 05:18:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 48, 736031), True, datetime.datetime(2023, 1, 17, 5, 18, 48, 736031), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:18:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:18:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:39.030814+00:00 (in 50.291879 seconds) repositorygcworker stdout | 2023-01-17 05:18:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:48 UTC)" (scheduled at 2023-01-17 05:18:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:18:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:18:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:18:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:18:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:24.641816+00:00 (in 35.811085 seconds) exportactionlogsworker stdout | 2023-01-17 05:18:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:48 UTC)" (scheduled at 2023-01-17 05:18:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:18:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:18:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 48, 831127), True, datetime.datetime(2023, 1, 17, 5, 18, 48, 831127), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:18:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:18:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:18:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:48 UTC)" executed successfully builder stdout | 2023-01-17 05:18:48,919 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:48,927 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:48,927 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:48,933 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:52,626 [243] [DEBUG] [app] Starting request: urn:request:455cc2b1-8320-46a6-9016-61559891fe60 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:52,626 [244] [DEBUG] [app] Starting request: urn:request:0cc54345-5fb6-435d-a71f-ecfed4e52219 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:18:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,628 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:52,631 [236] [DEBUG] [app] Starting request: urn:request:f1c9ae58-d471-4943-8344-127979687cea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,631 [240] [DEBUG] [app] Starting request: urn:request:1bd79820-ec22-426b-9406-d5fefc8c99ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,631 [236] [DEBUG] [app] Ending request: urn:request:f1c9ae58-d471-4943-8344-127979687cea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1c9ae58-d471-4943-8344-127979687cea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,631 [240] [DEBUG] [app] Ending request: urn:request:1bd79820-ec22-426b-9406-d5fefc8c99ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1bd79820-ec22-426b-9406-d5fefc8c99ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:18:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:52,636 [239] [DEBUG] [app] Starting request: urn:request:6d2413f8-1831-4b7a-8e18-52855532a1bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:52,637 [244] [DEBUG] [app] Starting request: urn:request:bbfb7104-caef-4fc5-925d-ea6fa2c51525 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:52,637 [239] [DEBUG] [app] Ending request: urn:request:6d2413f8-1831-4b7a-8e18-52855532a1bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d2413f8-1831-4b7a-8e18-52855532a1bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:52,637 [244] [DEBUG] [app] Ending request: urn:request:bbfb7104-caef-4fc5-925d-ea6fa2c51525 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bbfb7104-caef-4fc5-925d-ea6fa2c51525', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:18:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:52,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:52,648 [244] [DEBUG] [app] Ending request: urn:request:0cc54345-5fb6-435d-a71f-ecfed4e52219 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0cc54345-5fb6-435d-a71f-ecfed4e52219', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:52,648 [243] [DEBUG] [app] Ending request: urn:request:455cc2b1-8320-46a6-9016-61559891fe60 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:455cc2b1-8320-46a6-9016-61559891fe60', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:18:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:18:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:09.230038+00:00 (in 16.392207 seconds) securityworker stdout | 2023-01-17 05:18:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:22 UTC)" (scheduled at 2023-01-17 05:18:52.837365+00:00) securityworker stdout | 2023-01-17 05:18:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:58.735066+00:00 (in 1.799592 seconds) notificationworker stdout | 2023-01-17 05:18:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:23:56 UTC)" (scheduled at 2023-01-17 05:18:56.935055+00:00) notificationworker stdout | 2023-01-17 05:18:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 18, 56, 935839), 'notification/%']) notificationworker stdout | 2023-01-17 05:18:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 18, 56, 935839), True, datetime.datetime(2023, 1, 17, 5, 18, 56, 935839), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:18:56,946 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 18, 56, 935839), True, datetime.datetime(2023, 1, 17, 5, 18, 56, 935839), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 18, 56, 935839), 'notification/%']) notificationworker stdout | 2023-01-17 05:18:56,948 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:56,948 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:23:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:18:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:18:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:27.226424+00:00 (in 29.999540 seconds) buildlogsarchiver stdout | 2023-01-17 05:18:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:27 UTC)" (scheduled at 2023-01-17 05:18:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:18:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 18, 57, 227234), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:18:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:18:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:18:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:08.735066+00:00 (in 9.999581 seconds) notificationworker stdout | 2023-01-17 05:18:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:08 UTC)" (scheduled at 2023-01-17 05:18:58.735066+00:00) notificationworker stdout | 2023-01-17 05:18:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 58, 735908), True, datetime.datetime(2023, 1, 17, 5, 18, 58, 735908), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:58,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:08 UTC)" executed successfully builder stdout | 2023-01-17 05:18:58,943 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:58,952 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:58,952 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:58,958 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:18:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:18:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:15.931994+00:00 (in 16.594168 seconds) namespacegcworker stdout | 2023-01-17 05:18:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:59 UTC)" (scheduled at 2023-01-17 05:18:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:18:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:18:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 59, 338223), True, datetime.datetime(2023, 1, 17, 5, 18, 59, 338223), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:18:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:18:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:18:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:18:59,824 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:19:00,045 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:19:00,325 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:19:01,083 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:19:01,104 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:19:02,338 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:19:03,120 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:19:03,129 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:19:03,547 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:19:04,443 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:19:04,525 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:19:05,493 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:19:05,741 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:19:05,743 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:19:05,746 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:19:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:19:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:35.848675+00:00 (in 29.999454 seconds) gcworker stdout | 2023-01-17 05:19:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:35 UTC)" (scheduled at 2023-01-17 05:19:05.848675+00:00) gcworker stdout | 2023-01-17 05:19:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723145849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:19:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:19:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:19:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:19:06,244 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:19:06,282 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:19:07,451 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:19:07,625 [239] [DEBUG] [app] Starting request: urn:request:4f27ecf3-6c49-469e-8f12-920d2b4cc7c6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:07,625 [244] [DEBUG] [app] Starting request: urn:request:bad04231-5122-43a0-8461-417bbec46579 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:07,631 [232] [DEBUG] [app] Starting request: urn:request:56001f1c-8605-48bd-ab95-45771da22277 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,631 [237] [DEBUG] [app] Starting request: urn:request:1d17ff65-9d41-4630-9fde-cad990f2ec28 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,631 [232] [DEBUG] [app] Ending request: urn:request:56001f1c-8605-48bd-ab95-45771da22277 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56001f1c-8605-48bd-ab95-45771da22277', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,631 [237] [DEBUG] [app] Ending request: urn:request:1d17ff65-9d41-4630-9fde-cad990f2ec28 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d17ff65-9d41-4630-9fde-cad990f2ec28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:19:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:19:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:07,636 [243] [DEBUG] [app] Starting request: urn:request:26714062-4849-49c2-837f-3dcdf6734fff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:07,636 [241] [DEBUG] [app] Starting request: urn:request:4548e9e7-5c72-451d-9125-4d48b7a3b75a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:07,636 [243] [DEBUG] [app] Ending request: urn:request:26714062-4849-49c2-837f-3dcdf6734fff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26714062-4849-49c2-837f-3dcdf6734fff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:07,637 [241] [DEBUG] [app] Ending request: urn:request:4548e9e7-5c72-451d-9125-4d48b7a3b75a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4548e9e7-5c72-451d-9125-4d48b7a3b75a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:19:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:07,648 [239] [DEBUG] [app] Ending request: urn:request:4f27ecf3-6c49-469e-8f12-920d2b4cc7c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f27ecf3-6c49-469e-8f12-920d2b4cc7c6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:07,648 [244] [DEBUG] [app] Ending request: urn:request:bad04231-5122-43a0-8461-417bbec46579 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bad04231-5122-43a0-8461-417bbec46579', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:19:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) notificationworker stdout | 2023-01-17 05:19:08,009 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:19:08,074 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:19:08,077 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:19:08,080 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:19:08,083 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:19:08,085 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:19:08,087 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:19:08,093 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:19:08,097 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:19:08,099 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:19:08,320 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:19:08,324 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:19:08,327 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:19:08,330 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:19:08,332 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:19:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:18.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:19:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:18 UTC)" (scheduled at 2023-01-17 05:19:08.735066+00:00) notificationworker stdout | 2023-01-17 05:19:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 8, 735996), True, datetime.datetime(2023, 1, 17, 5, 19, 8, 735996), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:19:08,757 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:19:08,838 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:19:08,892 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:19:08,913 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:19:08,958 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:08,966 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:08,966 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:08,971 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:22.837365+00:00 (in 13.606879 seconds) securityworker stdout | 2023-01-17 05:19:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:39 UTC)" (scheduled at 2023-01-17 05:19:09.230038+00:00) securityworker stdout | 2023-01-17 05:19:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:19:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:19:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:19:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:39 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:19:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:19:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:47.932490+00:00 (in 32.000048 seconds) namespacegcworker stdout | 2023-01-17 05:19:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:24:15 UTC)" (scheduled at 2023-01-17 05:19:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:19:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 19, 15, 932765), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:19:15,939 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 19, 15, 932765), True, datetime.datetime(2023, 1, 17, 5, 19, 15, 932765), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:19:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 19, 15, 932765), True, datetime.datetime(2023, 1, 17, 5, 19, 15, 932765), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 19, 15, 932765), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:19:15,943 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:19:15,943 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:24:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:28.735066+00:00 (in 9.999356 seconds) notificationworker stdout | 2023-01-17 05:19:18,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:28 UTC)" (scheduled at 2023-01-17 05:19:18.735066+00:00) notificationworker stdout | 2023-01-17 05:19:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 18, 736193), True, datetime.datetime(2023, 1, 17, 5, 19, 18, 736193), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:28 UTC)" executed successfully builder stdout | 2023-01-17 05:19:18,978 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:18,986 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:18,986 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:18,992 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:19:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:19:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:25.334138+00:00 (in 5.091431 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:19:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:20 UTC)" (scheduled at 2023-01-17 05:19:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:19:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:19:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 20, 243106), True, datetime.datetime(2023, 1, 17, 5, 19, 20, 243106), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:19:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:19:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:19:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:19:22,625 [241] [DEBUG] [app] Starting request: urn:request:4d23b3ab-0e7f-48e9-9af6-67ed58c4db3c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:22,625 [244] [DEBUG] [app] Starting request: urn:request:49bcabcb-e99d-4f40-a420-3b2fa55bd761 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:22,630 [232] [DEBUG] [app] Starting request: urn:request:0b2aa22d-fb47-4d56-bf71-2cb02e93dc9b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,631 [242] [DEBUG] [app] Starting request: urn:request:a20a6e1f-cf28-42c6-9a5f-9ff51dd5e93c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,631 [232] [DEBUG] [app] Ending request: urn:request:0b2aa22d-fb47-4d56-bf71-2cb02e93dc9b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b2aa22d-fb47-4d56-bf71-2cb02e93dc9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,631 [242] [DEBUG] [app] Ending request: urn:request:a20a6e1f-cf28-42c6-9a5f-9ff51dd5e93c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a20a6e1f-cf28-42c6-9a5f-9ff51dd5e93c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:19:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:19:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:22,636 [243] [DEBUG] [app] Starting request: urn:request:e29edb0f-43fa-4a6d-adf3-ab91b04bc7d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:22,636 [239] [DEBUG] [app] Starting request: urn:request:b919055f-3be1-46d2-8332-39c78876572d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:22,636 [243] [DEBUG] [app] Ending request: urn:request:e29edb0f-43fa-4a6d-adf3-ab91b04bc7d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e29edb0f-43fa-4a6d-adf3-ab91b04bc7d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:22,637 [239] [DEBUG] [app] Ending request: urn:request:b919055f-3be1-46d2-8332-39c78876572d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b919055f-3be1-46d2-8332-39c78876572d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:19:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:22,648 [244] [DEBUG] [app] Ending request: urn:request:49bcabcb-e99d-4f40-a420-3b2fa55bd761 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:49bcabcb-e99d-4f40-a420-3b2fa55bd761', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:22,648 [241] [DEBUG] [app] Ending request: urn:request:4d23b3ab-0e7f-48e9-9af6-67ed58c4db3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d23b3ab-0e7f-48e9-9af6-67ed58c4db3c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:19:22,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 05:19:22,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:19:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:39.230038+00:00 (in 16.392231 seconds) securityworker stdout | 2023-01-17 05:19:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:52 UTC)" (scheduled at 2023-01-17 05:19:22.837365+00:00) securityworker stdout | 2023-01-17 05:19:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:19:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:19:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:48.830274+00:00 (in 24.188016 seconds) exportactionlogsworker stdout | 2023-01-17 05:19:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:24 UTC)" (scheduled at 2023-01-17 05:19:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:19:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:19:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:19:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:19:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:20.242198+00:00 (in 54.907559 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:19:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:25 UTC)" (scheduled at 2023-01-17 05:19:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:19:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:19:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:19:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:19:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:57.226424+00:00 (in 29.999533 seconds) buildlogsarchiver stdout | 2023-01-17 05:19:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:57 UTC)" (scheduled at 2023-01-17 05:19:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:19:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 19, 27, 227247), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:19:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:19:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:19:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:38.735066+00:00 (in 9.999454 seconds) notificationworker stdout | 2023-01-17 05:19:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:38 UTC)" (scheduled at 2023-01-17 05:19:28.735066+00:00) notificationworker stdout | 2023-01-17 05:19:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 28, 736117), True, datetime.datetime(2023, 1, 17, 5, 19, 28, 736117), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:38 UTC)" executed successfully builder stdout | 2023-01-17 05:19:29,001 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:29,009 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:29,009 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:29,015 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:29,860 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:19:30,069 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:19:30,345 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:19:31,099 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:19:31,122 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:19:32,369 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:19:33,152 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:19:33,158 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:19:33,582 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:19:34,479 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:19:34,560 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:19:35,509 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:19:35,748 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:19:35,751 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:19:35,753 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:19:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:19:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:05.848675+00:00 (in 29.999437 seconds) gcworker stdout | 2023-01-17 05:19:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:05 UTC)" (scheduled at 2023-01-17 05:19:35.848675+00:00) gcworker stdout | 2023-01-17 05:19:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723175849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:19:35,867 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:19:35,867 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:19:35,867 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:19:36,252 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:19:36,297 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:19:37,469 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:19:37,625 [239] [DEBUG] [app] Starting request: urn:request:ffa35396-1d2f-4274-b7fa-6153e3b10329 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:37,625 [244] [DEBUG] [app] Starting request: urn:request:9c63ccd8-aaeb-4ebd-a489-c6a58809aaa3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:37,630 [242] [DEBUG] [app] Starting request: urn:request:1acebe3b-7fa1-4bf3-a856-bc2f445b9b74 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:37,630 [240] [DEBUG] [app] Starting request: urn:request:11224524-f80d-4111-97b1-9f7d487eb9e0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:37,631 [242] [DEBUG] [app] Ending request: urn:request:1acebe3b-7fa1-4bf3-a856-bc2f445b9b74 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1acebe3b-7fa1-4bf3-a856-bc2f445b9b74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:19:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:19:37,631 [240] [DEBUG] [app] Ending request: urn:request:11224524-f80d-4111-97b1-9f7d487eb9e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11224524-f80d-4111-97b1-9f7d487eb9e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:19:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:37,635 [243] [DEBUG] [app] Starting request: urn:request:be52ff44-179c-4206-b112-1fd4078fcfce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:37,635 [241] [DEBUG] [app] Starting request: urn:request:d6b3fb62-625e-4d3b-a537-41195c5c534b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:37,635 [243] [DEBUG] [app] Ending request: urn:request:be52ff44-179c-4206-b112-1fd4078fcfce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be52ff44-179c-4206-b112-1fd4078fcfce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:37,636 [241] [DEBUG] [app] Ending request: urn:request:d6b3fb62-625e-4d3b-a537-41195c5c534b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d6b3fb62-625e-4d3b-a537-41195c5c534b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:19:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:37,646 [239] [DEBUG] [app] Ending request: urn:request:ffa35396-1d2f-4274-b7fa-6153e3b10329 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ffa35396-1d2f-4274-b7fa-6153e3b10329', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:37,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:37,646 [244] [DEBUG] [app] Ending request: urn:request:9c63ccd8-aaeb-4ebd-a489-c6a58809aaa3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c63ccd8-aaeb-4ebd-a489-c6a58809aaa3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:19:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:19:38,044 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:19:38,082 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:19:38,087 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:19:38,089 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:19:38,092 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:19:38,094 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:19:38,096 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:19:38,100 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:19:38,103 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:19:38,107 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:19:38,327 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:19:38,330 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:19:38,334 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:19:38,336 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:19:38,339 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:19:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:43.935629+00:00 (in 5.200052 seconds) notificationworker stdout | 2023-01-17 05:19:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:48 UTC)" (scheduled at 2023-01-17 05:19:38.735066+00:00) notificationworker stdout | 2023-01-17 05:19:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 38, 736013), True, datetime.datetime(2023, 1, 17, 5, 19, 38, 736013), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:48 UTC)" executed successfully gcworker stdout | 2023-01-17 05:19:38,773 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:19:38,873 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:19:38,920 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:19:38,932 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:19:39,025 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue repositorygcworker stdout | 2023-01-17 05:19:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:19:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:48.738458+00:00 (in 9.707234 seconds) repositorygcworker stdout | 2023-01-17 05:19:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:39 UTC)" (scheduled at 2023-01-17 05:19:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:19:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:19:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 39, 31625), True, datetime.datetime(2023, 1, 17, 5, 19, 39, 31625), 0, 'repositorygc/%', 50, 1, 0]) builder stdout | 2023-01-17 05:19:39,033 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:39,033 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics repositorygcworker stdout | 2023-01-17 05:19:39,038 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:19:39,038 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:19:39,038 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:39 UTC)" executed successfully builder stdout | 2023-01-17 05:19:39,039 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:52.837365+00:00 (in 13.606864 seconds) securityworker stdout | 2023-01-17 05:19:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:09 UTC)" (scheduled at 2023-01-17 05:19:39.230038+00:00) securityworker stdout | 2023-01-17 05:19:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:19:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:19:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:19:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:19:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:19:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:19:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:48.735066+00:00 (in 4.798943 seconds) notificationworker stdout | 2023-01-17 05:19:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:43 UTC)" (scheduled at 2023-01-17 05:19:43.935629+00:00) notificationworker stdout | 2023-01-17 05:19:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:19:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:19:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:19:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:59.337413+00:00 (in 11.404469 seconds) namespacegcworker stdout | 2023-01-17 05:19:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:47 UTC)" (scheduled at 2023-01-17 05:19:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:19:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:19:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:58.735066+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:19:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:58 UTC)" (scheduled at 2023-01-17 05:19:48.735066+00:00) notificationworker stdout | 2023-01-17 05:19:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 48, 735957), True, datetime.datetime(2023, 1, 17, 5, 19, 48, 735957), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:19:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:19:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:39.030814+00:00 (in 50.291909 seconds) repositorygcworker stdout | 2023-01-17 05:19:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:48 UTC)" (scheduled at 2023-01-17 05:19:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:19:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:19:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:19:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:19:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:24.641816+00:00 (in 35.811078 seconds) exportactionlogsworker stdout | 2023-01-17 05:19:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:48 UTC)" (scheduled at 2023-01-17 05:19:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:19:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:19:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 48, 831158), True, datetime.datetime(2023, 1, 17, 5, 19, 48, 831158), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:19:48,841 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:19:48,841 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:19:48,841 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:48 UTC)" executed successfully builder stdout | 2023-01-17 05:19:49,048 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:49,056 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:49,056 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:49,062 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:52,625 [239] [DEBUG] [app] Starting request: urn:request:3c7c8a5f-e65a-46fa-8013-606662cd23de (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:52,625 [244] [DEBUG] [app] Starting request: urn:request:5e26d273-ada4-4114-9d5f-48600552d003 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:19:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:52,630 [240] [DEBUG] [app] Starting request: urn:request:e245d325-4d48-4fa7-af55-eb09102bc274 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,630 [232] [DEBUG] [app] Starting request: urn:request:96ec41e5-f66f-4e5d-9cc5-a3e9d8c2d54e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,630 [240] [DEBUG] [app] Ending request: urn:request:e245d325-4d48-4fa7-af55-eb09102bc274 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e245d325-4d48-4fa7-af55-eb09102bc274', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,630 [232] [DEBUG] [app] Ending request: urn:request:96ec41e5-f66f-4e5d-9cc5-a3e9d8c2d54e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96ec41e5-f66f-4e5d-9cc5-a3e9d8c2d54e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:19:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:19:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:52,635 [243] [DEBUG] [app] Starting request: urn:request:045ea7b7-1391-427f-9f40-d1413cb01477 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:52,635 [241] [DEBUG] [app] Starting request: urn:request:433a1051-a020-419b-aff6-3884f2c3eb13 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:52,635 [243] [DEBUG] [app] Ending request: urn:request:045ea7b7-1391-427f-9f40-d1413cb01477 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:045ea7b7-1391-427f-9f40-d1413cb01477', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:52,636 [241] [DEBUG] [app] Ending request: urn:request:433a1051-a020-419b-aff6-3884f2c3eb13 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:433a1051-a020-419b-aff6-3884f2c3eb13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:19:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:52,647 [244] [DEBUG] [app] Ending request: urn:request:5e26d273-ada4-4114-9d5f-48600552d003 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5e26d273-ada4-4114-9d5f-48600552d003', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:52,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:19:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:19:52,648 [239] [DEBUG] [app] Ending request: urn:request:3c7c8a5f-e65a-46fa-8013-606662cd23de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c7c8a5f-e65a-46fa-8013-606662cd23de', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:19:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:09.230038+00:00 (in 16.392209 seconds) securityworker stdout | 2023-01-17 05:19:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:22 UTC)" (scheduled at 2023-01-17 05:19:52.837365+00:00) securityworker stdout | 2023-01-17 05:19:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:19:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:19:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:27.226424+00:00 (in 29.999553 seconds) buildlogsarchiver stdout | 2023-01-17 05:19:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:27 UTC)" (scheduled at 2023-01-17 05:19:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:19:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 19, 57, 227255), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:19:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:19:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:19:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:08.735066+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:19:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:08 UTC)" (scheduled at 2023-01-17 05:19:58.735066+00:00) notificationworker stdout | 2023-01-17 05:19:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 58, 736032), True, datetime.datetime(2023, 1, 17, 5, 19, 58, 736032), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:08 UTC)" executed successfully builder stdout | 2023-01-17 05:19:59,072 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:59,080 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:59,080 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:59,085 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:19:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:19:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:47.932490+00:00 (in 48.594598 seconds) namespacegcworker stdout | 2023-01-17 05:19:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:59 UTC)" (scheduled at 2023-01-17 05:19:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:19:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:19:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 59, 338282), True, datetime.datetime(2023, 1, 17, 5, 19, 59, 338282), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:19:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:19:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:19:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:19:59,896 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:20:00,085 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:20:00,351 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:20:01,131 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:20:01,147 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:20:02,404 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:20:03,187 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:20:03,193 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:20:03,605 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:20:04,508 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:20:04,565 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:20:05,520 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:20:05,756 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:20:05,759 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:20:05,761 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:20:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:20:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:35.848675+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 05:20:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:35 UTC)" (scheduled at 2023-01-17 05:20:05.848675+00:00) gcworker stdout | 2023-01-17 05:20:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723205849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:20:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:20:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:20:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:20:06,287 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:20:06,329 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:20:07,499 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:20:07,626 [239] [DEBUG] [app] Starting request: urn:request:0300363b-f0ff-41f1-89df-3e3ab17c8b17 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:07,626 [244] [DEBUG] [app] Starting request: urn:request:fcc83ebb-3181-4f03-a168-c1aee394cad1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:07,631 [232] [DEBUG] [app] Starting request: urn:request:5631ec4f-dcd9-4ab5-9f7a-c49d29fd1cdb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:07,631 [240] [DEBUG] [app] Starting request: urn:request:3eb235b7-937d-4252-85e5-655606723105 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:07,632 [232] [DEBUG] [app] Ending request: urn:request:5631ec4f-dcd9-4ab5-9f7a-c49d29fd1cdb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5631ec4f-dcd9-4ab5-9f7a-c49d29fd1cdb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:07,632 [240] [DEBUG] [app] Ending request: urn:request:3eb235b7-937d-4252-85e5-655606723105 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3eb235b7-937d-4252-85e5-655606723105', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:20:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:07,636 [243] [DEBUG] [app] Starting request: urn:request:bf29599e-aa50-4912-a75d-b094dad3cb29 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:07,636 [244] [DEBUG] [app] Starting request: urn:request:9cf18e98-3033-4dd0-8b83-a96615feb089 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:07,637 [243] [DEBUG] [app] Ending request: urn:request:bf29599e-aa50-4912-a75d-b094dad3cb29 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf29599e-aa50-4912-a75d-b094dad3cb29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:07,637 [244] [DEBUG] [app] Ending request: urn:request:9cf18e98-3033-4dd0-8b83-a96615feb089 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9cf18e98-3033-4dd0-8b83-a96615feb089', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:20:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:07,648 [239] [DEBUG] [app] Ending request: urn:request:0300363b-f0ff-41f1-89df-3e3ab17c8b17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0300363b-f0ff-41f1-89df-3e3ab17c8b17', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:07,648 [244] [DEBUG] [app] Ending request: urn:request:fcc83ebb-3181-4f03-a168-c1aee394cad1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fcc83ebb-3181-4f03-a168-c1aee394cad1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:20:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) notificationworker stdout | 2023-01-17 05:20:08,080 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:20:08,089 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:20:08,095 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:20:08,097 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:20:08,100 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:20:08,103 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:20:08,105 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:20:08,108 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:20:08,112 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:20:08,114 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:20:08,333 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:20:08,336 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:20:08,340 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:20:08,343 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:20:08,345 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:20:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:18.735066+00:00 (in 9.999369 seconds) notificationworker stdout | 2023-01-17 05:20:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:18 UTC)" (scheduled at 2023-01-17 05:20:08.735066+00:00) notificationworker stdout | 2023-01-17 05:20:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 8, 736112), True, datetime.datetime(2023, 1, 17, 5, 20, 8, 736112), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:20:08,785 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:20:08,881 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:20:08,955 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:20:08,958 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:20:09,091 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:09,099 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:09,099 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:09,105 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:22.837365+00:00 (in 13.606903 seconds) securityworker stdout | 2023-01-17 05:20:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:39 UTC)" (scheduled at 2023-01-17 05:20:09.230038+00:00) securityworker stdout | 2023-01-17 05:20:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:20:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:20:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:20:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:28.735066+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:20:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:28 UTC)" (scheduled at 2023-01-17 05:20:18.735066+00:00) notificationworker stdout | 2023-01-17 05:20:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 18, 735982), True, datetime.datetime(2023, 1, 17, 5, 20, 18, 735982), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:28 UTC)" executed successfully builder stdout | 2023-01-17 05:20:19,105 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:19,114 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:19,114 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:19,120 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:20:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:20:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:25.334138+00:00 (in 5.091486 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:20:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:20 UTC)" (scheduled at 2023-01-17 05:20:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:20:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:20:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 20, 243026), True, datetime.datetime(2023, 1, 17, 5, 20, 20, 243026), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:20:20,252 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:20:20,252 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:20:20,252 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:20:22,625 [239] [DEBUG] [app] Starting request: urn:request:84e15ce6-0dc0-40ae-af5f-b9a9fca2c538 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:22,625 [243] [DEBUG] [app] Starting request: urn:request:63ddf31e-926b-4eb4-88a9-ae2478de74d0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:22,630 [232] [DEBUG] [app] Starting request: urn:request:f314cdd0-435f-4c70-8835-6ac0f6cf8529 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,631 [242] [DEBUG] [app] Starting request: urn:request:7009aa19-7250-4626-a28b-2cfb09334d36 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,631 [232] [DEBUG] [app] Ending request: urn:request:f314cdd0-435f-4c70-8835-6ac0f6cf8529 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f314cdd0-435f-4c70-8835-6ac0f6cf8529', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,631 [242] [DEBUG] [app] Ending request: urn:request:7009aa19-7250-4626-a28b-2cfb09334d36 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7009aa19-7250-4626-a28b-2cfb09334d36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:20:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:20:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:22,636 [244] [DEBUG] [app] Starting request: urn:request:f48d0fd6-ef49-45ed-9830-5b007bc194fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:22,636 [239] [DEBUG] [app] Starting request: urn:request:0ae8be21-2d45-4e0b-929e-964d5960862f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:22,636 [244] [DEBUG] [app] Ending request: urn:request:f48d0fd6-ef49-45ed-9830-5b007bc194fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f48d0fd6-ef49-45ed-9830-5b007bc194fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:22,636 [239] [DEBUG] [app] Ending request: urn:request:0ae8be21-2d45-4e0b-929e-964d5960862f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0ae8be21-2d45-4e0b-929e-964d5960862f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:22,648 [239] [DEBUG] [app] Ending request: urn:request:84e15ce6-0dc0-40ae-af5f-b9a9fca2c538 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84e15ce6-0dc0-40ae-af5f-b9a9fca2c538', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:22,648 [243] [DEBUG] [app] Ending request: urn:request:63ddf31e-926b-4eb4-88a9-ae2478de74d0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63ddf31e-926b-4eb4-88a9-ae2478de74d0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:20:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:20:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:39.230038+00:00 (in 16.392260 seconds) securityworker stdout | 2023-01-17 05:20:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:52 UTC)" (scheduled at 2023-01-17 05:20:22.837365+00:00) securityworker stdout | 2023-01-17 05:20:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:20:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:20:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:48.830274+00:00 (in 24.187975 seconds) exportactionlogsworker stdout | 2023-01-17 05:20:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:24 UTC)" (scheduled at 2023-01-17 05:20:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:20:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:20:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:20:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:20:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:20.242198+00:00 (in 54.907622 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:20:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:25 UTC)" (scheduled at 2023-01-17 05:20:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:20:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:20:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:20:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:20:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:57.226424+00:00 (in 29.999551 seconds) buildlogsarchiver stdout | 2023-01-17 05:20:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:57 UTC)" (scheduled at 2023-01-17 05:20:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:20:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 20, 27, 227208), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:20:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:20:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:20:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:38.735066+00:00 (in 9.999470 seconds) notificationworker stdout | 2023-01-17 05:20:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:38 UTC)" (scheduled at 2023-01-17 05:20:28.735066+00:00) notificationworker stdout | 2023-01-17 05:20:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 28, 736095), True, datetime.datetime(2023, 1, 17, 5, 20, 28, 736095), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:38 UTC)" executed successfully builder stdout | 2023-01-17 05:20:29,130 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:29,139 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:29,139 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:29,145 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:29,933 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:20:30,090 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:20:30,357 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:20:31,155 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:20:31,177 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:20:32,440 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:20:33,196 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:20:33,226 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:20:33,630 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:20:34,544 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:20:34,572 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:20:35,535 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:20:35,763 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:20:35,766 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:20:35,768 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:20:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:20:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:05.848675+00:00 (in 29.999543 seconds) gcworker stdout | 2023-01-17 05:20:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:05 UTC)" (scheduled at 2023-01-17 05:20:35.848675+00:00) gcworker stdout | 2023-01-17 05:20:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723235849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:20:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:20:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:20:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:20:36,294 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:20:36,348 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:20:37,535 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:20:37,626 [243] [DEBUG] [app] Starting request: urn:request:0d6619f4-d630-4a9c-a1b1-e4d4d9cfda12 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:37,626 [241] [DEBUG] [app] Starting request: urn:request:cee99fcb-aa74-4cc8-8811-5e6cad1c3ba6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:37,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:37,631 [240] [DEBUG] [app] Starting request: urn:request:24dc29fb-bdf6-45c9-87f6-56e26daf0d62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,632 [240] [DEBUG] [app] Ending request: urn:request:24dc29fb-bdf6-45c9-87f6-56e26daf0d62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24dc29fb-bdf6-45c9-87f6-56e26daf0d62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,632 [232] [DEBUG] [app] Starting request: urn:request:e264a97d-023c-4f72-b179-732b8282e260 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:20:37,632 [232] [DEBUG] [app] Ending request: urn:request:e264a97d-023c-4f72-b179-732b8282e260 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e264a97d-023c-4f72-b179-732b8282e260', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,633 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:20:37,633 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:37,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,635 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:37,637 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:37,637 [239] [DEBUG] [app] Starting request: urn:request:5c4ef229-17ed-4abf-9484-a78470f7f4eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:37,637 [239] [DEBUG] [app] Ending request: urn:request:5c4ef229-17ed-4abf-9484-a78470f7f4eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5c4ef229-17ed-4abf-9484-a78470f7f4eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:37,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:37,638 [244] [DEBUG] [app] Starting request: urn:request:3e241b64-d99b-4ec5-9f12-2e38f4f7a8cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:37,638 [244] [DEBUG] [app] Ending request: urn:request:3e241b64-d99b-4ec5-9f12-2e38f4f7a8cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e241b64-d99b-4ec5-9f12-2e38f4f7a8cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:37,639 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:37,639 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:37,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:37,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:37,644 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:37,649 [243] [DEBUG] [app] Ending request: urn:request:0d6619f4-d630-4a9c-a1b1-e4d4d9cfda12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d6619f4-d630-4a9c-a1b1-e4d4d9cfda12', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:37,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:37,649 [241] [DEBUG] [app] Ending request: urn:request:cee99fcb-aa74-4cc8-8811-5e6cad1c3ba6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cee99fcb-aa74-4cc8-8811-5e6cad1c3ba6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:37,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:37,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:20:37,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-registry stdout | 2023-01-17 05:20:38,097 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} notificationworker stdout | 2023-01-17 05:20:38,099 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:20:38,104 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:20:38,107 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:20:38,109 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:20:38,112 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:20:38,114 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:20:38,116 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:20:38,120 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:20:38,122 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:20:38,341 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:20:38,343 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:20:38,346 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:20:38,350 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:20:38,352 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:20:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:43.935629+00:00 (in 5.200046 seconds) notificationworker stdout | 2023-01-17 05:20:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:48 UTC)" (scheduled at 2023-01-17 05:20:38.735066+00:00) notificationworker stdout | 2023-01-17 05:20:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 38, 736007), True, datetime.datetime(2023, 1, 17, 5, 20, 38, 736007), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:48 UTC)" executed successfully gcworker stdout | 2023-01-17 05:20:38,807 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:20:38,889 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:20:38,973 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:20:38,975 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} repositorygcworker stdout | 2023-01-17 05:20:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:20:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:48.738458+00:00 (in 9.707250 seconds) repositorygcworker stdout | 2023-01-17 05:20:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:39 UTC)" (scheduled at 2023-01-17 05:20:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:20:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:20:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 39, 31595), True, datetime.datetime(2023, 1, 17, 5, 20, 39, 31595), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:20:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:20:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:20:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:39 UTC)" executed successfully builder stdout | 2023-01-17 05:20:39,155 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:39,164 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:39,164 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:39,169 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:52.837365+00:00 (in 13.606907 seconds) securityworker stdout | 2023-01-17 05:20:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:09 UTC)" (scheduled at 2023-01-17 05:20:39.230038+00:00) securityworker stdout | 2023-01-17 05:20:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:20:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:20:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:20:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:48.735066+00:00 (in 4.798937 seconds) notificationworker stdout | 2023-01-17 05:20:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:43 UTC)" (scheduled at 2023-01-17 05:20:43.935629+00:00) notificationworker stdout | 2023-01-17 05:20:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:20:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:20:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:20:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:59.337413+00:00 (in 11.404447 seconds) namespacegcworker stdout | 2023-01-17 05:20:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:47 UTC)" (scheduled at 2023-01-17 05:20:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:20:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:20:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:58.735066+00:00 (in 9.999424 seconds) notificationworker stdout | 2023-01-17 05:20:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:58 UTC)" (scheduled at 2023-01-17 05:20:48.735066+00:00) notificationworker stdout | 2023-01-17 05:20:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 48, 736092), True, datetime.datetime(2023, 1, 17, 5, 20, 48, 736092), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:20:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:20:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:39.030814+00:00 (in 50.291926 seconds) repositorygcworker stdout | 2023-01-17 05:20:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:48 UTC)" (scheduled at 2023-01-17 05:20:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:20:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:20:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:20:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:20:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:24.641816+00:00 (in 35.811117 seconds) exportactionlogsworker stdout | 2023-01-17 05:20:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:48 UTC)" (scheduled at 2023-01-17 05:20:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:20:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:20:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 48, 831064), True, datetime.datetime(2023, 1, 17, 5, 20, 48, 831064), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:20:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:20:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:20:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:48 UTC)" executed successfully builder stdout | 2023-01-17 05:20:49,180 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:49,188 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:49,188 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:49,194 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:52,625 [243] [DEBUG] [app] Starting request: urn:request:9fc977bb-6836-4b3f-b789-1332c3486e17 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:52,625 [244] [DEBUG] [app] Starting request: urn:request:8ff00774-c810-44dd-8e2d-4812d3453c60 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:20:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:52,630 [232] [DEBUG] [app] Starting request: urn:request:d848f8a1-786e-42aa-86d8-a5395b2bfa14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,631 [240] [DEBUG] [app] Starting request: urn:request:c09f3f76-80a8-4017-bc76-47fcdd68329f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,631 [232] [DEBUG] [app] Ending request: urn:request:d848f8a1-786e-42aa-86d8-a5395b2bfa14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d848f8a1-786e-42aa-86d8-a5395b2bfa14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,631 [240] [DEBUG] [app] Ending request: urn:request:c09f3f76-80a8-4017-bc76-47fcdd68329f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c09f3f76-80a8-4017-bc76-47fcdd68329f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:20:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:52,636 [239] [DEBUG] [app] Starting request: urn:request:31d76b60-1fc6-4618-83f8-ad766332d82f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:52,636 [244] [DEBUG] [app] Starting request: urn:request:03a44452-29d1-4317-90b6-b7ed77a50f84 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:52,636 [239] [DEBUG] [app] Ending request: urn:request:31d76b60-1fc6-4618-83f8-ad766332d82f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31d76b60-1fc6-4618-83f8-ad766332d82f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:52,637 [244] [DEBUG] [app] Ending request: urn:request:03a44452-29d1-4317-90b6-b7ed77a50f84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03a44452-29d1-4317-90b6-b7ed77a50f84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:20:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:52,648 [244] [DEBUG] [app] Ending request: urn:request:8ff00774-c810-44dd-8e2d-4812d3453c60 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ff00774-c810-44dd-8e2d-4812d3453c60', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:52,648 [243] [DEBUG] [app] Ending request: urn:request:9fc977bb-6836-4b3f-b789-1332c3486e17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9fc977bb-6836-4b3f-b789-1332c3486e17', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:20:52,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:20:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:20:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:09.230038+00:00 (in 16.392199 seconds) securityworker stdout | 2023-01-17 05:20:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:22 UTC)" (scheduled at 2023-01-17 05:20:52.837365+00:00) securityworker stdout | 2023-01-17 05:20:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:52,844 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:20:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:20:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:27.226424+00:00 (in 29.999542 seconds) buildlogsarchiver stdout | 2023-01-17 05:20:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:27 UTC)" (scheduled at 2023-01-17 05:20:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:20:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 20, 57, 227230), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:20:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:20:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:20:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:08.735066+00:00 (in 9.999499 seconds) notificationworker stdout | 2023-01-17 05:20:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:08 UTC)" (scheduled at 2023-01-17 05:20:58.735066+00:00) notificationworker stdout | 2023-01-17 05:20:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 58, 736053), True, datetime.datetime(2023, 1, 17, 5, 20, 58, 736053), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:08 UTC)" executed successfully builder stdout | 2023-01-17 05:20:59,194 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:59,203 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:59,203 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:59,209 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:20:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:20:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:47.932490+00:00 (in 48.594609 seconds) namespacegcworker stdout | 2023-01-17 05:20:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:59 UTC)" (scheduled at 2023-01-17 05:20:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:20:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:20:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 59, 338276), True, datetime.datetime(2023, 1, 17, 5, 20, 59, 338276), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:20:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:20:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:20:59,345 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:20:59,940 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:21:00,114 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:21:00,375 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:21:01,182 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:21:01,207 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:21:02,475 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:21:03,226 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:21:03,255 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:21:03,638 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:21:04,551 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:21:04,596 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:21:05,568 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:21:05,771 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:21:05,774 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:21:05,776 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:21:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:21:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:35.848675+00:00 (in 29.999562 seconds) gcworker stdout | 2023-01-17 05:21:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:35 UTC)" (scheduled at 2023-01-17 05:21:05.848675+00:00) gcworker stdout | 2023-01-17 05:21:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723265849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:21:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:21:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:21:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:21:06,325 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:21:06,383 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:21:07,541 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:21:07,625 [239] [DEBUG] [app] Starting request: urn:request:0f64d78c-0315-487b-bfe4-a1e0367f7be0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:07,625 [244] [DEBUG] [app] Starting request: urn:request:bac87cfa-fc46-4c4c-abe8-326ad7379238 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:07,630 [240] [DEBUG] [app] Starting request: urn:request:60928120-c9df-4f51-b31b-4e83c41a8eb8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:07,630 [240] [DEBUG] [app] Ending request: urn:request:60928120-c9df-4f51-b31b-4e83c41a8eb8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60928120-c9df-4f51-b31b-4e83c41a8eb8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:07,630 [232] [DEBUG] [app] Starting request: urn:request:b1681060-55eb-4db3-823f-1d386919f417 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:21:07,631 [232] [DEBUG] [app] Ending request: urn:request:b1681060-55eb-4db3-823f-1d386919f417 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1681060-55eb-4db3-823f-1d386919f417', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:21:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:21:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:07,635 [244] [DEBUG] [app] Starting request: urn:request:fd0288e4-4db4-454f-9c56-9fefb84066c5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:07,635 [244] [DEBUG] [app] Ending request: urn:request:fd0288e4-4db4-454f-9c56-9fefb84066c5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd0288e4-4db4-454f-9c56-9fefb84066c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:07,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:07,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:07,637 [243] [DEBUG] [app] Starting request: urn:request:8b07ad3b-0c6d-4ecc-87ab-bdb9092fe216 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:07,637 [243] [DEBUG] [app] Ending request: urn:request:8b07ad3b-0c6d-4ecc-87ab-bdb9092fe216 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b07ad3b-0c6d-4ecc-87ab-bdb9092fe216', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:07,646 [239] [DEBUG] [app] Ending request: urn:request:0f64d78c-0315-487b-bfe4-a1e0367f7be0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f64d78c-0315-487b-bfe4-a1e0367f7be0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:21:07,647 [244] [DEBUG] [app] Ending request: urn:request:bac87cfa-fc46-4c4c-abe8-326ad7379238 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bac87cfa-fc46-4c4c-abe8-326ad7379238', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-registry stdout | 2023-01-17 05:21:08,106 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:21:08,112 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:21:08,117 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} notificationworker stdout | 2023-01-17 05:21:08,119 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-registry stdout | 2023-01-17 05:21:08,122 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:21:08,124 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:21:08,126 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:21:08,129 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:21:08,131 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:21:08,134 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:21:08,347 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:21:08,350 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:21:08,352 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:21:08,355 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:21:08,358 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:21:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:18.735066+00:00 (in 9.999491 seconds) notificationworker stdout | 2023-01-17 05:21:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:18 UTC)" (scheduled at 2023-01-17 05:21:08.735066+00:00) notificationworker stdout | 2023-01-17 05:21:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 8, 735995), True, datetime.datetime(2023, 1, 17, 5, 21, 8, 735995), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:21:08,834 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:21:08,925 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:21:08,980 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:21:09,002 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:21:09,219 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:09,228 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:09,228 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics securityworker stdout | 2023-01-17 05:21:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:22.837365+00:00 (in 13.606904 seconds) securityworker stdout | 2023-01-17 05:21:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:39 UTC)" (scheduled at 2023-01-17 05:21:09.230038+00:00) securityworker stdout | 2023-01-17 05:21:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:21:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:21:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state builder stdout | 2023-01-17 05:21:09,234 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:09,240 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:09,240 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:21:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:28.735066+00:00 (in 9.999484 seconds) notificationworker stdout | 2023-01-17 05:21:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:28 UTC)" (scheduled at 2023-01-17 05:21:18.735066+00:00) notificationworker stdout | 2023-01-17 05:21:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:18,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 18, 736008), True, datetime.datetime(2023, 1, 17, 5, 21, 18, 736008), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:28 UTC)" executed successfully builder stdout | 2023-01-17 05:21:19,244 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:19,252 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:19,252 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:19,257 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:21:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:21:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:25.334138+00:00 (in 5.091493 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:21:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:20 UTC)" (scheduled at 2023-01-17 05:21:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:21:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:21:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 20, 243059), True, datetime.datetime(2023, 1, 17, 5, 21, 20, 243059), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:21:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:21:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:21:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:21:22,625 [244] [DEBUG] [app] Starting request: urn:request:dc45c1c2-828d-4e59-a307-e738f570c77f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:22,625 [241] [DEBUG] [app] Starting request: urn:request:f9dd651e-a007-4f64-bbe6-6689c0a07b0c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:22,630 [232] [DEBUG] [app] Starting request: urn:request:eb86ae95-75e7-4cba-a17d-a1e2e42d09c7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:22,631 [242] [DEBUG] [app] Starting request: urn:request:5adab2eb-1fe0-408f-a41f-716ce5c46bc1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:22,631 [232] [DEBUG] [app] Ending request: urn:request:eb86ae95-75e7-4cba-a17d-a1e2e42d09c7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb86ae95-75e7-4cba-a17d-a1e2e42d09c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:21:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:21:22,631 [242] [DEBUG] [app] Ending request: urn:request:5adab2eb-1fe0-408f-a41f-716ce5c46bc1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5adab2eb-1fe0-408f-a41f-716ce5c46bc1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:21:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:21:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:22,636 [239] [DEBUG] [app] Starting request: urn:request:7ac9fc2f-d989-4ef5-b6a9-cef147e4023c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:22,636 [243] [DEBUG] [app] Starting request: urn:request:1df75028-250d-4841-b9bb-8343c2038035 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:22,636 [239] [DEBUG] [app] Ending request: urn:request:7ac9fc2f-d989-4ef5-b6a9-cef147e4023c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ac9fc2f-d989-4ef5-b6a9-cef147e4023c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:22,637 [243] [DEBUG] [app] Ending request: urn:request:1df75028-250d-4841-b9bb-8343c2038035 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1df75028-250d-4841-b9bb-8343c2038035', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:21:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:22,647 [241] [DEBUG] [app] Ending request: urn:request:f9dd651e-a007-4f64-bbe6-6689c0a07b0c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f9dd651e-a007-4f64-bbe6-6689c0a07b0c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:22,647 [244] [DEBUG] [app] Ending request: urn:request:dc45c1c2-828d-4e59-a307-e738f570c77f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc45c1c2-828d-4e59-a307-e738f570c77f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:22,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:21:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:21:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:39.230038+00:00 (in 16.392266 seconds) securityworker stdout | 2023-01-17 05:21:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:52 UTC)" (scheduled at 2023-01-17 05:21:22.837365+00:00) securityworker stdout | 2023-01-17 05:21:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:21:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:21:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:44.641497+00:00 (in 19.999261 seconds) exportactionlogsworker stdout | 2023-01-17 05:21:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:24 UTC)" (scheduled at 2023-01-17 05:21:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:21:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:21:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:21:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:21:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:39.333860+00:00 (in 13.999249 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:21:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:25 UTC)" (scheduled at 2023-01-17 05:21:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:21:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:21:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:21:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:21:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:57.226424+00:00 (in 29.999521 seconds) buildlogsarchiver stdout | 2023-01-17 05:21:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:57 UTC)" (scheduled at 2023-01-17 05:21:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:21:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 21, 27, 227260), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:21:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:21:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:21:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:38.735066+00:00 (in 9.999494 seconds) notificationworker stdout | 2023-01-17 05:21:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:38 UTC)" (scheduled at 2023-01-17 05:21:28.735066+00:00) notificationworker stdout | 2023-01-17 05:21:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 28, 736066), True, datetime.datetime(2023, 1, 17, 5, 21, 28, 736066), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:38 UTC)" executed successfully builder stdout | 2023-01-17 05:21:29,268 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:29,276 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:29,276 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:29,282 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:29,955 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:21:30,135 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:21:30,386 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:21:31,217 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:21:31,243 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:21:32,501 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:21:33,263 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:21:33,266 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:21:33,667 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:21:34,586 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:21:34,607 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:21:35,576 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:21:35,779 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:21:35,782 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:21:35,784 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:21:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:21:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:05.848675+00:00 (in 29.999549 seconds) gcworker stdout | 2023-01-17 05:21:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:05 UTC)" (scheduled at 2023-01-17 05:21:35.848675+00:00) gcworker stdout | 2023-01-17 05:21:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723295849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:21:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:21:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:21:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:21:36,341 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:21:36,417 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:21:37,570 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:21:37,625 [239] [DEBUG] [app] Starting request: urn:request:26336245-ba7d-4088-8766-c246a94a9045 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:37,625 [243] [DEBUG] [app] Starting request: urn:request:6cf0c921-2fab-418d-9d0d-a414e517d888 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:37,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:37,630 [232] [DEBUG] [app] Starting request: urn:request:436c69ac-3901-4ae6-9bb8-0d05ff1594d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:37,630 [242] [DEBUG] [app] Starting request: urn:request:07cd2f25-8c29-4892-aa78-df68e56ffbaf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:37,630 [232] [DEBUG] [app] Ending request: urn:request:436c69ac-3901-4ae6-9bb8-0d05ff1594d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:436c69ac-3901-4ae6-9bb8-0d05ff1594d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:21:37,631 [242] [DEBUG] [app] Ending request: urn:request:07cd2f25-8c29-4892-aa78-df68e56ffbaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07cd2f25-8c29-4892-aa78-df68e56ffbaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:21:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:37,635 [241] [DEBUG] [app] Starting request: urn:request:53b8e279-4f60-49a1-a57e-7c202acfa021 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:37,635 [244] [DEBUG] [app] Starting request: urn:request:ea394cb7-2c56-42c2-aec4-2ff856a32d22 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:37,636 [241] [DEBUG] [app] Ending request: urn:request:53b8e279-4f60-49a1-a57e-7c202acfa021 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53b8e279-4f60-49a1-a57e-7c202acfa021', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:37,636 [244] [DEBUG] [app] Ending request: urn:request:ea394cb7-2c56-42c2-aec4-2ff856a32d22 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea394cb7-2c56-42c2-aec4-2ff856a32d22', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:21:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:21:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:37,647 [239] [DEBUG] [app] Ending request: urn:request:26336245-ba7d-4088-8766-c246a94a9045 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26336245-ba7d-4088-8766-c246a94a9045', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:21:37,651 [243] [DEBUG] [app] Ending request: urn:request:6cf0c921-2fab-418d-9d0d-a414e517d888 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cf0c921-2fab-418d-9d0d-a414e517d888', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:37,651 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.027) gunicorn-web stdout | 2023-01-17 05:21:37,651 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:21:38,116 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:21:38,120 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:21:38,125 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:21:38,130 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:21:38,133 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:21:38,135 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:21:38,138 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:21:38,140 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:21:38,143 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:21:38,153 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:21:38,353 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:21:38,356 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:21:38,358 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:21:38,362 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:21:38,364 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:21:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:43.935629+00:00 (in 5.200033 seconds) notificationworker stdout | 2023-01-17 05:21:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:48 UTC)" (scheduled at 2023-01-17 05:21:38.735066+00:00) notificationworker stdout | 2023-01-17 05:21:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 38, 736022), True, datetime.datetime(2023, 1, 17, 5, 21, 38, 736022), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:48 UTC)" executed successfully gcworker stdout | 2023-01-17 05:21:38,849 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:21:38,932 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:21:38,991 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:21:39,011 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} repositorygcworker stdout | 2023-01-17 05:21:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:21:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:48.738458+00:00 (in 9.707225 seconds) repositorygcworker stdout | 2023-01-17 05:21:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:39 UTC)" (scheduled at 2023-01-17 05:21:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:21:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:21:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 39, 31629), True, datetime.datetime(2023, 1, 17, 5, 21, 39, 31629), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:21:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:21:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:21:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:21:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:52.837365+00:00 (in 13.606918 seconds) securityworker stdout | 2023-01-17 05:21:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:09 UTC)" (scheduled at 2023-01-17 05:21:39.230038+00:00) securityworker stdout | 2023-01-17 05:21:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:21:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:21:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:21:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:09 UTC)" executed successfully builder stdout | 2023-01-17 05:21:39,282 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:39,290 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:39,290 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:39,296 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:21:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:21:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:20.242198+00:00 (in 40.907905 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:21:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:39 UTC)" (scheduled at 2023-01-17 05:21:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:21:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 21, 39, 334650), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:21:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 39, 334650), True, datetime.datetime(2023, 1, 17, 5, 21, 39, 334650), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:21:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 39, 334650), True, datetime.datetime(2023, 1, 17, 5, 21, 39, 334650), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 21, 39, 334650), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:21:39,344 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:21:39,344 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:48.735066+00:00 (in 4.798979 seconds) notificationworker stdout | 2023-01-17 05:21:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:43 UTC)" (scheduled at 2023-01-17 05:21:43.935629+00:00) notificationworker stdout | 2023-01-17 05:21:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:21:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:21:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:21:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:48.830274+00:00 (in 4.188277 seconds) exportactionlogsworker stdout | 2023-01-17 05:21:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:44 UTC)" (scheduled at 2023-01-17 05:21:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:21:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 21, 44, 642376), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:21:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 44, 642376), True, datetime.datetime(2023, 1, 17, 5, 21, 44, 642376), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:21:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 44, 642376), True, datetime.datetime(2023, 1, 17, 5, 21, 44, 642376), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 21, 44, 642376), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:21:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:21:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:21:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:21:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:59.337413+00:00 (in 11.404452 seconds) namespacegcworker stdout | 2023-01-17 05:21:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:47 UTC)" (scheduled at 2023-01-17 05:21:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:21:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:21:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:58.735066+00:00 (in 9.999486 seconds) notificationworker stdout | 2023-01-17 05:21:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:58 UTC)" (scheduled at 2023-01-17 05:21:48.735066+00:00) notificationworker stdout | 2023-01-17 05:21:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 48, 736036), True, datetime.datetime(2023, 1, 17, 5, 21, 48, 736036), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:21:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:21:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:04.738062+00:00 (in 15.999143 seconds) repositorygcworker stdout | 2023-01-17 05:21:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:48 UTC)" (scheduled at 2023-01-17 05:21:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:21:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:21:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:21:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:21:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:24.641816+00:00 (in 35.811098 seconds) exportactionlogsworker stdout | 2023-01-17 05:21:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:48 UTC)" (scheduled at 2023-01-17 05:21:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:21:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:21:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 48, 831085), True, datetime.datetime(2023, 1, 17, 5, 21, 48, 831085), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:21:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:21:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:21:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:48 UTC)" executed successfully builder stdout | 2023-01-17 05:21:49,306 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:49,314 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:49,314 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:49,320 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:52,625 [239] [DEBUG] [app] Starting request: urn:request:7d4db5da-3046-4f99-9cc1-a5f277103b38 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:52,625 [243] [DEBUG] [app] Starting request: urn:request:286d99ee-4c22-4f83-9fa6-803872ac8f51 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:21:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:52,630 [240] [DEBUG] [app] Starting request: urn:request:8b8229b0-caf5-4992-929d-a1c0996c62ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:52,631 [236] [DEBUG] [app] Starting request: urn:request:927f54c5-ba90-44dc-81c2-aab1237dc218 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:52,631 [240] [DEBUG] [app] Ending request: urn:request:8b8229b0-caf5-4992-929d-a1c0996c62ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b8229b0-caf5-4992-929d-a1c0996c62ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:21:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:21:52,631 [236] [DEBUG] [app] Ending request: urn:request:927f54c5-ba90-44dc-81c2-aab1237dc218 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:927f54c5-ba90-44dc-81c2-aab1237dc218', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:21:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:52,636 [244] [DEBUG] [app] Starting request: urn:request:ddd5a51f-ec91-40ed-8bd0-224947fcb86c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:52,636 [239] [DEBUG] [app] Starting request: urn:request:92983ad4-1830-44f6-97e7-f3c40e63c879 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:52,637 [244] [DEBUG] [app] Ending request: urn:request:ddd5a51f-ec91-40ed-8bd0-224947fcb86c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ddd5a51f-ec91-40ed-8bd0-224947fcb86c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:52,637 [239] [DEBUG] [app] Ending request: urn:request:92983ad4-1830-44f6-97e7-f3c40e63c879 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92983ad4-1830-44f6-97e7-f3c40e63c879', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:52,646 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:52,648 [243] [DEBUG] [app] Ending request: urn:request:286d99ee-4c22-4f83-9fa6-803872ac8f51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:286d99ee-4c22-4f83-9fa6-803872ac8f51', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:21:52,654 [239] [DEBUG] [app] Ending request: urn:request:7d4db5da-3046-4f99-9cc1-a5f277103b38 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d4db5da-3046-4f99-9cc1-a5f277103b38', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:52,654 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:52,654 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityworker stdout | 2023-01-17 05:21:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:09.230038+00:00 (in 16.392225 seconds) securityworker stdout | 2023-01-17 05:21:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:22 UTC)" (scheduled at 2023-01-17 05:21:52.837365+00:00) securityworker stdout | 2023-01-17 05:21:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:21:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:21:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:27.226424+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 05:21:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:27 UTC)" (scheduled at 2023-01-17 05:21:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:21:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 21, 57, 227287), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:21:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:21:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:21:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:08.735066+00:00 (in 9.999440 seconds) notificationworker stdout | 2023-01-17 05:21:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:08 UTC)" (scheduled at 2023-01-17 05:21:58.735066+00:00) notificationworker stdout | 2023-01-17 05:21:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 58, 736102), True, datetime.datetime(2023, 1, 17, 5, 21, 58, 736102), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:08 UTC)" executed successfully builder stdout | 2023-01-17 05:21:59,321 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:59,330 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:59,330 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:59,335 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:21:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:21:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:47.932490+00:00 (in 48.594644 seconds) namespacegcworker stdout | 2023-01-17 05:21:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:59 UTC)" (scheduled at 2023-01-17 05:21:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:21:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:21:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 59, 338198), True, datetime.datetime(2023, 1, 17, 5, 21, 59, 338198), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:21:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:21:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:21:59,345 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:21:59,991 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:22:00,149 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:22:00,422 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:22:01,253 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:22:01,277 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:22:02,537 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:22:03,274 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:22:03,298 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:22:03,703 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:22:04,622 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:22:04,642 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} repositorygcworker stdout | 2023-01-17 05:22:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:22:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:39.030814+00:00 (in 34.292324 seconds) repositorygcworker stdout | 2023-01-17 05:22:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:27:04 UTC)" (scheduled at 2023-01-17 05:22:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:22:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 22, 4, 738858), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:22:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 22, 4, 738858), True, datetime.datetime(2023, 1, 17, 5, 22, 4, 738858), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:22:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 22, 4, 738858), True, datetime.datetime(2023, 1, 17, 5, 22, 4, 738858), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 22, 4, 738858), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:22:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:22:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:27:04 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:22:05,612 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:22:05,787 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:22:05,789 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:22:05,792 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:22:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:22:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:35.848675+00:00 (in 29.999560 seconds) gcworker stdout | 2023-01-17 05:22:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:35 UTC)" (scheduled at 2023-01-17 05:22:05.848675+00:00) gcworker stdout | 2023-01-17 05:22:05,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:22:05,860 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723325860, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:22:05,863 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:22:05,863 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:22:05,863 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:22:06,365 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:22:06,428 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:22:07,590 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:22:07,625 [239] [DEBUG] [app] Starting request: urn:request:5af75dce-0d7b-4d78-afbc-4604d3ab17d9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:07,625 [244] [DEBUG] [app] Starting request: urn:request:50cb1227-f672-4314-a41a-40fdcf058ae2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:07,630 [232] [DEBUG] [app] Starting request: urn:request:42db8aa0-07da-44db-8c2e-412e2fed119c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:07,631 [232] [DEBUG] [app] Ending request: urn:request:42db8aa0-07da-44db-8c2e-412e2fed119c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:42db8aa0-07da-44db-8c2e-412e2fed119c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:07,631 [242] [DEBUG] [app] Starting request: urn:request:edfa25ea-d38f-43a5-b2fe-00ceaeffc116 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:22:07,631 [242] [DEBUG] [app] Ending request: urn:request:edfa25ea-d38f-43a5-b2fe-00ceaeffc116 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:edfa25ea-d38f-43a5-b2fe-00ceaeffc116', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:22:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:07,636 [244] [DEBUG] [app] Starting request: urn:request:1752b5df-4462-4a09-92c9-3aae9575076c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:07,636 [244] [DEBUG] [app] Ending request: urn:request:1752b5df-4462-4a09-92c9-3aae9575076c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1752b5df-4462-4a09-92c9-3aae9575076c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:07,636 [241] [DEBUG] [app] Starting request: urn:request:cffb50f8-0ca8-4df0-9e92-7ed35d06c035 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:07,637 [241] [DEBUG] [app] Ending request: urn:request:cffb50f8-0ca8-4df0-9e92-7ed35d06c035 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cffb50f8-0ca8-4df0-9e92-7ed35d06c035', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:22:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:07,648 [244] [DEBUG] [app] Ending request: urn:request:50cb1227-f672-4314-a41a-40fdcf058ae2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50cb1227-f672-4314-a41a-40fdcf058ae2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:07,648 [239] [DEBUG] [app] Ending request: urn:request:5af75dce-0d7b-4d78-afbc-4604d3ab17d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5af75dce-0d7b-4d78-afbc-4604d3ab17d9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:22:07,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:22:08,125 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:22:08,127 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:22:08,130 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:22:08,137 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:22:08,140 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:22:08,144 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:22:08,146 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:22:08,149 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:22:08,152 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:22:08,187 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:22:08,359 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:22:08,362 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:22:08,366 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:22:08,368 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:22:08,371 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:22:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:18.735066+00:00 (in 9.999393 seconds) notificationworker stdout | 2023-01-17 05:22:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:18 UTC)" (scheduled at 2023-01-17 05:22:08.735066+00:00) notificationworker stdout | 2023-01-17 05:22:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 8, 736043), True, datetime.datetime(2023, 1, 17, 5, 22, 8, 736043), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:22:08,885 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:22:08,967 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:22:09,017 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:22:09,046 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:22:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:22.837365+00:00 (in 13.606870 seconds) securityworker stdout | 2023-01-17 05:22:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:39 UTC)" (scheduled at 2023-01-17 05:22:09.230038+00:00) securityworker stdout | 2023-01-17 05:22:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:22:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:22:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:22:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:39 UTC)" executed successfully builder stdout | 2023-01-17 05:22:09,340 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:09,348 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:09,348 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:09,354 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:28.735066+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:22:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:28 UTC)" (scheduled at 2023-01-17 05:22:18.735066+00:00) notificationworker stdout | 2023-01-17 05:22:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 18, 735950), True, datetime.datetime(2023, 1, 17, 5, 22, 18, 735950), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:28 UTC)" executed successfully builder stdout | 2023-01-17 05:22:19,364 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:19,373 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:19,373 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:19,378 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:22:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:22:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:25.334138+00:00 (in 5.091482 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:22:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:20 UTC)" (scheduled at 2023-01-17 05:22:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:22:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:22:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 20, 243066), True, datetime.datetime(2023, 1, 17, 5, 22, 20, 243066), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:22:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:22:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:22:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:22:22,625 [239] [DEBUG] [app] Starting request: urn:request:ddf22407-cb56-4c3a-9add-7c19cbdfecb2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:22,625 [244] [DEBUG] [app] Starting request: urn:request:162736fc-2544-4800-83d4-8ab2a57920a0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:22,630 [232] [DEBUG] [app] Starting request: urn:request:9e9e5f29-dd77-48b0-8996-f7debc27257d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:22,631 [232] [DEBUG] [app] Ending request: urn:request:9e9e5f29-dd77-48b0-8996-f7debc27257d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e9e5f29-dd77-48b0-8996-f7debc27257d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:22,631 [240] [DEBUG] [app] Starting request: urn:request:202f399d-b5c3-4500-80ef-7d2323b26082 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:22:22,631 [240] [DEBUG] [app] Ending request: urn:request:202f399d-b5c3-4500-80ef-7d2323b26082 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:202f399d-b5c3-4500-80ef-7d2323b26082', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:22:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:22:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:22,635 [241] [DEBUG] [app] Starting request: urn:request:7c64621c-f0ba-46f3-b00f-606305c1dc4e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:22,635 [244] [DEBUG] [app] Starting request: urn:request:6167797f-7440-4906-b690-4de638d4ff17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:22,636 [241] [DEBUG] [app] Ending request: urn:request:7c64621c-f0ba-46f3-b00f-606305c1dc4e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c64621c-f0ba-46f3-b00f-606305c1dc4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:22,636 [244] [DEBUG] [app] Ending request: urn:request:6167797f-7440-4906-b690-4de638d4ff17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6167797f-7440-4906-b690-4de638d4ff17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:22,647 [244] [DEBUG] [app] Ending request: urn:request:162736fc-2544-4800-83d4-8ab2a57920a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:162736fc-2544-4800-83d4-8ab2a57920a0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:22,647 [239] [DEBUG] [app] Ending request: urn:request:ddf22407-cb56-4c3a-9add-7c19cbdfecb2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddf22407-cb56-4c3a-9add-7c19cbdfecb2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:22:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:22:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:39.230038+00:00 (in 16.392229 seconds) securityworker stdout | 2023-01-17 05:22:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:52 UTC)" (scheduled at 2023-01-17 05:22:22.837365+00:00) securityworker stdout | 2023-01-17 05:22:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:22:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:22:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:48.830274+00:00 (in 24.188008 seconds) exportactionlogsworker stdout | 2023-01-17 05:22:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:24 UTC)" (scheduled at 2023-01-17 05:22:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:22:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:22:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:22:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:22:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:20.242198+00:00 (in 54.907606 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:22:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:25 UTC)" (scheduled at 2023-01-17 05:22:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:22:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:22:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:22:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:22:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:57.226424+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 05:22:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:57 UTC)" (scheduled at 2023-01-17 05:22:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:22:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 22, 27, 227337), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:22:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:22:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:22:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:38.735066+00:00 (in 9.999503 seconds) notificationworker stdout | 2023-01-17 05:22:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:38 UTC)" (scheduled at 2023-01-17 05:22:28.735066+00:00) notificationworker stdout | 2023-01-17 05:22:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 28, 736004), True, datetime.datetime(2023, 1, 17, 5, 22, 28, 736004), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:38 UTC)" executed successfully builder stdout | 2023-01-17 05:22:29,388 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:29,397 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:29,397 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:29,403 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:30,003 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:22:30,157 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:22:30,430 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:22:31,290 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:22:31,292 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:22:32,573 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:22:33,310 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:22:33,331 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:22:33,739 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:22:34,655 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:22:34,657 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:22:35,620 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:22:35,795 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:22:35,797 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:22:35,800 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:22:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:22:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:05.848675+00:00 (in 29.999513 seconds) gcworker stdout | 2023-01-17 05:22:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:05 UTC)" (scheduled at 2023-01-17 05:22:35.848675+00:00) gcworker stdout | 2023-01-17 05:22:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723355849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:22:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:22:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:22:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:22:36,372 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:22:36,433 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:22:37,596 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:22:37,626 [239] [DEBUG] [app] Starting request: urn:request:7cc80921-3f15-4627-be0e-cfec8ca4348f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:37,626 [243] [DEBUG] [app] Starting request: urn:request:6e97b04e-8e3d-4a0b-9273-f5c995f82708 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:37,631 [242] [DEBUG] [app] Starting request: urn:request:3ac39288-d3a3-45a6-b193-45d0b84c4743 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,631 [240] [DEBUG] [app] Starting request: urn:request:fea9d1be-3c76-4a20-88e5-a2d7c57f023b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,631 [242] [DEBUG] [app] Ending request: urn:request:3ac39288-d3a3-45a6-b193-45d0b84c4743 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ac39288-d3a3-45a6-b193-45d0b84c4743', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,632 [240] [DEBUG] [app] Ending request: urn:request:fea9d1be-3c76-4a20-88e5-a2d7c57f023b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fea9d1be-3c76-4a20-88e5-a2d7c57f023b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:22:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:22:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:37,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:37,636 [244] [DEBUG] [app] Starting request: urn:request:e8b4f7a5-18bc-486e-8ad1-d2c77d8ac570 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:37,637 [239] [DEBUG] [app] Starting request: urn:request:e4383997-adfd-4bd6-a16b-82f36bed83a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:37,637 [244] [DEBUG] [app] Ending request: urn:request:e8b4f7a5-18bc-486e-8ad1-d2c77d8ac570 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e8b4f7a5-18bc-486e-8ad1-d2c77d8ac570', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:37,637 [239] [DEBUG] [app] Ending request: urn:request:e4383997-adfd-4bd6-a16b-82f36bed83a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4383997-adfd-4bd6-a16b-82f36bed83a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:37,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:37,648 [239] [DEBUG] [app] Ending request: urn:request:7cc80921-3f15-4627-be0e-cfec8ca4348f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7cc80921-3f15-4627-be0e-cfec8ca4348f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:22:37,648 [243] [DEBUG] [app] Ending request: urn:request:6e97b04e-8e3d-4a0b-9273-f5c995f82708 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e97b04e-8e3d-4a0b-9273-f5c995f82708', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:37,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:37,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-registry stdout | 2023-01-17 05:22:38,134 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:22:38,136 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:22:38,138 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:22:38,143 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:22:38,147 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:22:38,151 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:22:38,154 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:22:38,156 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:22:38,161 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:22:38,222 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:22:38,366 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:22:38,368 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:22:38,371 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:22:38,373 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:22:38,376 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:22:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:43.935629+00:00 (in 5.200062 seconds) notificationworker stdout | 2023-01-17 05:22:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:48 UTC)" (scheduled at 2023-01-17 05:22:38.735066+00:00) notificationworker stdout | 2023-01-17 05:22:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 38, 735978), True, datetime.datetime(2023, 1, 17, 5, 22, 38, 735978), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:48 UTC)" executed successfully gcworker stdout | 2023-01-17 05:22:38,914 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:22:38,998 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} repositorygcworker stdout | 2023-01-17 05:22:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:22:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:48.738458+00:00 (in 9.707202 seconds) repositorygcworker stdout | 2023-01-17 05:22:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:39 UTC)" (scheduled at 2023-01-17 05:22:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:22:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:22:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 39, 31685), True, datetime.datetime(2023, 1, 17, 5, 22, 39, 31685), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:22:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:22:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:22:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:39 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:22:39,052 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:22:39,081 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:22:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:52.837365+00:00 (in 13.606670 seconds) securityworker stdout | 2023-01-17 05:22:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:09 UTC)" (scheduled at 2023-01-17 05:22:39.230038+00:00) securityworker stdout | 2023-01-17 05:22:39,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:22:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:22:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:22:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:22:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:22:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:22:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:09 UTC)" executed successfully builder stdout | 2023-01-17 05:22:39,404 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:39,412 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:39,412 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:39,418 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:48.735066+00:00 (in 4.798962 seconds) notificationworker stdout | 2023-01-17 05:22:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:43 UTC)" (scheduled at 2023-01-17 05:22:43.935629+00:00) notificationworker stdout | 2023-01-17 05:22:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:22:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:22:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:22:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:59.337413+00:00 (in 11.404465 seconds) namespacegcworker stdout | 2023-01-17 05:22:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:47 UTC)" (scheduled at 2023-01-17 05:22:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:22:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:22:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:58.735066+00:00 (in 9.999590 seconds) notificationworker stdout | 2023-01-17 05:22:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:58 UTC)" (scheduled at 2023-01-17 05:22:48.735066+00:00) notificationworker stdout | 2023-01-17 05:22:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 48, 735855), True, datetime.datetime(2023, 1, 17, 5, 22, 48, 735855), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:22:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:22:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:39.030814+00:00 (in 50.291892 seconds) repositorygcworker stdout | 2023-01-17 05:22:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:48 UTC)" (scheduled at 2023-01-17 05:22:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:22:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:22:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:22:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:22:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:24.641816+00:00 (in 35.811052 seconds) exportactionlogsworker stdout | 2023-01-17 05:22:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:48 UTC)" (scheduled at 2023-01-17 05:22:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:22:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:22:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 48, 831160), True, datetime.datetime(2023, 1, 17, 5, 22, 48, 831160), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:22:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:22:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:22:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:48 UTC)" executed successfully builder stdout | 2023-01-17 05:22:49,428 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:49,436 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:49,436 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:49,442 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:52,626 [241] [DEBUG] [app] Starting request: urn:request:0d6fc1df-97ef-41c4-a4ed-8e70c89a34bc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:52,626 [244] [DEBUG] [app] Starting request: urn:request:8bdc81de-b7b4-44b5-b8cf-26874a59cd84 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:22:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:52,631 [242] [DEBUG] [app] Starting request: urn:request:4b066b29-2afe-4e38-9dfd-4dc488598a27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:52,631 [232] [DEBUG] [app] Starting request: urn:request:d0092c2a-ef71-4e68-959b-9a02e6d7a30d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:52,631 [242] [DEBUG] [app] Ending request: urn:request:4b066b29-2afe-4e38-9dfd-4dc488598a27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b066b29-2afe-4e38-9dfd-4dc488598a27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:22:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:22:52,631 [232] [DEBUG] [app] Ending request: urn:request:d0092c2a-ef71-4e68-959b-9a02e6d7a30d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d0092c2a-ef71-4e68-959b-9a02e6d7a30d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:22:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:52,636 [243] [DEBUG] [app] Starting request: urn:request:2f505e79-33e0-4d1f-8ba2-393e2aa53252 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:52,637 [239] [DEBUG] [app] Starting request: urn:request:a0913772-3b1b-49df-ac8d-4e9c52bc1a8c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:52,637 [243] [DEBUG] [app] Ending request: urn:request:2f505e79-33e0-4d1f-8ba2-393e2aa53252 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f505e79-33e0-4d1f-8ba2-393e2aa53252', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:22:52,637 [239] [DEBUG] [app] Ending request: urn:request:a0913772-3b1b-49df-ac8d-4e9c52bc1a8c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0913772-3b1b-49df-ac8d-4e9c52bc1a8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:52,648 [241] [DEBUG] [app] Ending request: urn:request:0d6fc1df-97ef-41c4-a4ed-8e70c89a34bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d6fc1df-97ef-41c4-a4ed-8e70c89a34bc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:52,648 [244] [DEBUG] [app] Ending request: urn:request:8bdc81de-b7b4-44b5-b8cf-26874a59cd84 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8bdc81de-b7b4-44b5-b8cf-26874a59cd84', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:22:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:22:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:09.230038+00:00 (in 16.392204 seconds) securityworker stdout | 2023-01-17 05:22:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:22 UTC)" (scheduled at 2023-01-17 05:22:52.837365+00:00) securityworker stdout | 2023-01-17 05:22:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:22:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:22:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:27.226424+00:00 (in 29.999518 seconds) buildlogsarchiver stdout | 2023-01-17 05:22:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:27 UTC)" (scheduled at 2023-01-17 05:22:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:22:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 22, 57, 227285), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:22:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:22:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:22:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:08.735066+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 05:22:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:08 UTC)" (scheduled at 2023-01-17 05:22:58.735066+00:00) notificationworker stdout | 2023-01-17 05:22:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 58, 736009), True, datetime.datetime(2023, 1, 17, 5, 22, 58, 736009), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:58,756 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:58,756 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:58,756 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:22:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:22:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:47.932490+00:00 (in 48.594569 seconds) namespacegcworker stdout | 2023-01-17 05:22:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:59 UTC)" (scheduled at 2023-01-17 05:22:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:22:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:22:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 59, 338357), True, datetime.datetime(2023, 1, 17, 5, 22, 59, 338357), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:22:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:22:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:22:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:59 UTC)" executed successfully builder stdout | 2023-01-17 05:22:59,452 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:59,461 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:59,461 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:59,466 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:00,036 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:23:00,192 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:23:00,465 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:23:01,300 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:23:01,302 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:23:02,606 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:23:03,332 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:23:03,343 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:23:03,747 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositoryactioncounter stdout | 2023-01-17 05:23:04,682 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:23:04,690 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:23:05,648 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:23:05,803 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:23:05,805 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:23:05,807 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:23:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:23:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:35.848675+00:00 (in 29.999556 seconds) gcworker stdout | 2023-01-17 05:23:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:35 UTC)" (scheduled at 2023-01-17 05:23:05.848675+00:00) gcworker stdout | 2023-01-17 05:23:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723385849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:23:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:23:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:23:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:23:06,394 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:23:06,466 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 05:23:07,607 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:23:07,625 [243] [DEBUG] [app] Starting request: urn:request:0d2ee41b-e274-4e29-aa59-3a114f12fdb4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:07,626 [244] [DEBUG] [app] Starting request: urn:request:c510515c-1493-4062-8980-3169ceae186a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:07,630 [242] [DEBUG] [app] Starting request: urn:request:236ebd2b-0257-491e-8dc2-fd29fec5f7a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,630 [232] [DEBUG] [app] Starting request: urn:request:461cea63-e852-4b97-9d84-82633c85ae55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,631 [242] [DEBUG] [app] Ending request: urn:request:236ebd2b-0257-491e-8dc2-fd29fec5f7a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:236ebd2b-0257-491e-8dc2-fd29fec5f7a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,631 [232] [DEBUG] [app] Ending request: urn:request:461cea63-e852-4b97-9d84-82633c85ae55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:461cea63-e852-4b97-9d84-82633c85ae55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:23:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:07,635 [243] [DEBUG] [app] Starting request: urn:request:b7db0bd9-12e6-4b99-9dc7-7972bdfcaa71 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:07,636 [243] [DEBUG] [app] Ending request: urn:request:b7db0bd9-12e6-4b99-9dc7-7972bdfcaa71 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7db0bd9-12e6-4b99-9dc7-7972bdfcaa71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:07,636 [241] [DEBUG] [app] Starting request: urn:request:290b04f1-ca6b-47b3-a262-f554c8fbafe4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:23:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:07,636 [241] [DEBUG] [app] Ending request: urn:request:290b04f1-ca6b-47b3-a262-f554c8fbafe4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:290b04f1-ca6b-47b3-a262-f554c8fbafe4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:07,647 [243] [DEBUG] [app] Ending request: urn:request:0d2ee41b-e274-4e29-aa59-3a114f12fdb4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d2ee41b-e274-4e29-aa59-3a114f12fdb4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:07,647 [244] [DEBUG] [app] Ending request: urn:request:c510515c-1493-4062-8980-3169ceae186a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c510515c-1493-4062-8980-3169ceae186a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:23:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:23:08,143 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:23:08,146 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:23:08,149 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:23:08,151 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:23:08,153 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:23:08,157 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:23:08,161 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:23:08,164 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:23:08,169 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:23:08,257 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:23:08,373 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:23:08,375 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:23:08,378 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:23:08,380 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:23:08,383 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:23:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:18.735066+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:23:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:18 UTC)" (scheduled at 2023-01-17 05:23:08.735066+00:00) notificationworker stdout | 2023-01-17 05:23:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 8, 736014), True, datetime.datetime(2023, 1, 17, 5, 23, 8, 736014), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:23:08,933 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:23:09,033 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:23:09,087 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:23:09,116 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:23:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:22.837365+00:00 (in 13.606860 seconds) securityworker stdout | 2023-01-17 05:23:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:39 UTC)" (scheduled at 2023-01-17 05:23:09.230038+00:00) securityworker stdout | 2023-01-17 05:23:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:23:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:23:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:09,232 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:09,232 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:09,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:23:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:23:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:23:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:09,243 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:09,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:23:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:39 UTC)" executed successfully builder stdout | 2023-01-17 05:23:09,477 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:09,485 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:09,485 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:09,491 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:28.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 05:23:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:28 UTC)" (scheduled at 2023-01-17 05:23:18.735066+00:00) notificationworker stdout | 2023-01-17 05:23:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 18, 736044), True, datetime.datetime(2023, 1, 17, 5, 23, 18, 736044), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:28 UTC)" executed successfully builder stdout | 2023-01-17 05:23:19,494 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:19,502 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:19,502 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:19,508 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:23:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:23:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:25.334138+00:00 (in 5.091466 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:23:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:20 UTC)" (scheduled at 2023-01-17 05:23:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:23:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:23:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 20, 243096), True, datetime.datetime(2023, 1, 17, 5, 23, 20, 243096), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:23:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:23:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:23:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:23:22,625 [239] [DEBUG] [app] Starting request: urn:request:7e1b54c3-72e5-4fd0-b8cc-8cddab6461bc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:22,625 [244] [DEBUG] [app] Starting request: urn:request:cc1a0fee-a01a-4eeb-886e-d6a0b771d0b4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:22,630 [242] [DEBUG] [app] Starting request: urn:request:f7ce2828-7f13-482d-bc33-f279c6109e34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,631 [232] [DEBUG] [app] Starting request: urn:request:163ceb43-237f-4502-8287-e9862ba58f59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,631 [242] [DEBUG] [app] Ending request: urn:request:f7ce2828-7f13-482d-bc33-f279c6109e34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f7ce2828-7f13-482d-bc33-f279c6109e34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:23:22,631 [232] [DEBUG] [app] Ending request: urn:request:163ceb43-237f-4502-8287-e9862ba58f59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:163ceb43-237f-4502-8287-e9862ba58f59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:23:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:23:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:22,635 [241] [DEBUG] [app] Starting request: urn:request:beeb4081-e46f-4223-a4ee-2a17e825adf5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:22,635 [244] [DEBUG] [app] Starting request: urn:request:2ec04354-5462-45af-984c-705caac9fe83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:22,636 [241] [DEBUG] [app] Ending request: urn:request:beeb4081-e46f-4223-a4ee-2a17e825adf5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:beeb4081-e46f-4223-a4ee-2a17e825adf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:22,636 [244] [DEBUG] [app] Ending request: urn:request:2ec04354-5462-45af-984c-705caac9fe83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ec04354-5462-45af-984c-705caac9fe83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:22,646 [244] [DEBUG] [app] Ending request: urn:request:cc1a0fee-a01a-4eeb-886e-d6a0b771d0b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc1a0fee-a01a-4eeb-886e-d6a0b771d0b4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:22,647 [239] [DEBUG] [app] Ending request: urn:request:7e1b54c3-72e5-4fd0-b8cc-8cddab6461bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e1b54c3-72e5-4fd0-b8cc-8cddab6461bc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:23:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:23:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:39.230038+00:00 (in 16.392207 seconds) securityworker stdout | 2023-01-17 05:23:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:52 UTC)" (scheduled at 2023-01-17 05:23:22.837365+00:00) securityworker stdout | 2023-01-17 05:23:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:22,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:23:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:23:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:48.830274+00:00 (in 24.187983 seconds) exportactionlogsworker stdout | 2023-01-17 05:23:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:24 UTC)" (scheduled at 2023-01-17 05:23:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:23:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:23:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:23:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:23:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:20.242198+00:00 (in 54.907595 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:23:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:25 UTC)" (scheduled at 2023-01-17 05:23:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:23:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:23:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:23:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:23:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:57.226424+00:00 (in 29.999543 seconds) buildlogsarchiver stdout | 2023-01-17 05:23:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:57 UTC)" (scheduled at 2023-01-17 05:23:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:23:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 23, 27, 227235), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:23:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:23:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:23:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:38.735066+00:00 (in 9.999426 seconds) notificationworker stdout | 2023-01-17 05:23:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:38 UTC)" (scheduled at 2023-01-17 05:23:28.735066+00:00) notificationworker stdout | 2023-01-17 05:23:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 28, 736100), True, datetime.datetime(2023, 1, 17, 5, 23, 28, 736100), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:38 UTC)" executed successfully builder stdout | 2023-01-17 05:23:29,514 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:29,524 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:29,524 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:29,530 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:30,068 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:23:30,208 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:23:30,500 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:23:31,330 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:23:31,333 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:23:32,638 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:23:33,359 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:23:33,371 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:23:33,782 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:23:34,714 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:23:34,716 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:23:35,684 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:23:35,810 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:23:35,812 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:23:35,815 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:23:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:23:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:05.848675+00:00 (in 29.999592 seconds) gcworker stdout | 2023-01-17 05:23:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:05 UTC)" (scheduled at 2023-01-17 05:23:35.848675+00:00) gcworker stdout | 2023-01-17 05:23:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723415849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:23:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:23:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:23:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:23:36,430 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:23:36,473 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:23:37,625 [243] [DEBUG] [app] Starting request: urn:request:8d45add1-3d71-49a4-ab43-b8c8cf0d45b6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:37,626 [239] [DEBUG] [app] Starting request: urn:request:b5f3cee7-791e-4509-a9d6-11126eecee86 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:37,631 [232] [DEBUG] [app] Starting request: urn:request:dbf80abe-b0a7-4bee-95ab-88b0cec5d5b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:37,631 [242] [DEBUG] [app] Starting request: urn:request:8bdb961d-9ebc-49fe-9cbe-324906f0d4df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:37,631 [232] [DEBUG] [app] Ending request: urn:request:dbf80abe-b0a7-4bee-95ab-88b0cec5d5b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dbf80abe-b0a7-4bee-95ab-88b0cec5d5b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:23:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:23:37,632 [242] [DEBUG] [app] Ending request: urn:request:8bdb961d-9ebc-49fe-9cbe-324906f0d4df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8bdb961d-9ebc-49fe-9cbe-324906f0d4df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:23:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:23:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:37,636 [244] [DEBUG] [app] Starting request: urn:request:709f936e-c7c3-441f-939b-a4c56489bafb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:37,636 [244] [DEBUG] [app] Ending request: urn:request:709f936e-c7c3-441f-939b-a4c56489bafb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:709f936e-c7c3-441f-939b-a4c56489bafb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:37,636 [241] [DEBUG] [app] Starting request: urn:request:4779787d-dd0d-4521-abaf-8526244231ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:37,637 [241] [DEBUG] [app] Ending request: urn:request:4779787d-dd0d-4521-abaf-8526244231ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4779787d-dd0d-4521-abaf-8526244231ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:23:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) storagereplication stdout | 2023-01-17 05:23:37,642 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:23:37,647 [239] [DEBUG] [app] Ending request: urn:request:b5f3cee7-791e-4509-a9d6-11126eecee86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5f3cee7-791e-4509-a9d6-11126eecee86', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:37,647 [243] [DEBUG] [app] Ending request: urn:request:8d45add1-3d71-49a4-ab43-b8c8cf0d45b6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d45add1-3d71-49a4-ab43-b8c8cf0d45b6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:23:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-registry stdout | 2023-01-17 05:23:38,151 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:23:38,153 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:23:38,157 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:23:38,160 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:23:38,162 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:23:38,165 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:23:38,168 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:23:38,171 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:23:38,175 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:23:38,291 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:23:38,379 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:23:38,382 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:23:38,386 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:23:38,388 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:23:38,390 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:23:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:43.935629+00:00 (in 5.200065 seconds) notificationworker stdout | 2023-01-17 05:23:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:48 UTC)" (scheduled at 2023-01-17 05:23:38.735066+00:00) notificationworker stdout | 2023-01-17 05:23:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 38, 735926), True, datetime.datetime(2023, 1, 17, 5, 23, 38, 735926), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:48 UTC)" executed successfully gcworker stdout | 2023-01-17 05:23:38,959 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} repositorygcworker stdout | 2023-01-17 05:23:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:23:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:48.738458+00:00 (in 9.707209 seconds) repositorygcworker stdout | 2023-01-17 05:23:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:39 UTC)" (scheduled at 2023-01-17 05:23:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:23:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:23:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 39, 31664), True, datetime.datetime(2023, 1, 17, 5, 23, 39, 31664), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:23:39,049 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:23:39,049 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:23:39,049 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:39 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:23:39,059 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:23:39,095 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:23:39,121 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:23:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:52.837365+00:00 (in 13.606893 seconds) securityworker stdout | 2023-01-17 05:23:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:09 UTC)" (scheduled at 2023-01-17 05:23:39.230038+00:00) securityworker stdout | 2023-01-17 05:23:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:23:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:23:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:23:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:09 UTC)" executed successfully builder stdout | 2023-01-17 05:23:39,540 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:39,548 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:39,548 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:39,554 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:48.735066+00:00 (in 4.798995 seconds) notificationworker stdout | 2023-01-17 05:23:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:43 UTC)" (scheduled at 2023-01-17 05:23:43.935629+00:00) notificationworker stdout | 2023-01-17 05:23:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:23:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:23:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:23:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:59.337413+00:00 (in 11.404440 seconds) namespacegcworker stdout | 2023-01-17 05:23:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:47 UTC)" (scheduled at 2023-01-17 05:23:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:23:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:23:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:56.935055+00:00 (in 8.199503 seconds) notificationworker stdout | 2023-01-17 05:23:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:58 UTC)" (scheduled at 2023-01-17 05:23:48.735066+00:00) notificationworker stdout | 2023-01-17 05:23:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 48, 735937), True, datetime.datetime(2023, 1, 17, 5, 23, 48, 735937), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:23:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:23:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:39.030814+00:00 (in 50.291913 seconds) repositorygcworker stdout | 2023-01-17 05:23:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:48 UTC)" (scheduled at 2023-01-17 05:23:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:23:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:23:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:23:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:23:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:24.641816+00:00 (in 35.811097 seconds) exportactionlogsworker stdout | 2023-01-17 05:23:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:48 UTC)" (scheduled at 2023-01-17 05:23:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:23:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:23:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 48, 831056), True, datetime.datetime(2023, 1, 17, 5, 23, 48, 831056), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:23:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:23:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:23:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:48 UTC)" executed successfully builder stdout | 2023-01-17 05:23:49,560 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:49,568 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:49,568 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:49,574 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:52,625 [244] [DEBUG] [app] Starting request: urn:request:c19b6203-e215-4401-928f-3ecdc504e39b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:52,625 [239] [DEBUG] [app] Starting request: urn:request:a7f25a28-5261-46be-9ed9-51304cf1f51b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:23:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:52,631 [242] [DEBUG] [app] Starting request: urn:request:da7b1d54-f402-40d7-9b21-ca42646bb74e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,631 [232] [DEBUG] [app] Starting request: urn:request:054d1c6a-8cae-4b01-a237-12704bcdd9ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,631 [232] [DEBUG] [app] Ending request: urn:request:054d1c6a-8cae-4b01-a237-12704bcdd9ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:054d1c6a-8cae-4b01-a237-12704bcdd9ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,631 [242] [DEBUG] [app] Ending request: urn:request:da7b1d54-f402-40d7-9b21-ca42646bb74e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da7b1d54-f402-40d7-9b21-ca42646bb74e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:23:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:23:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:52,636 [243] [DEBUG] [app] Starting request: urn:request:945fc6bf-4e50-4351-a9b3-0440642513ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:52,636 [239] [DEBUG] [app] Starting request: urn:request:58039417-7090-408a-bf9c-2295dfa0d534 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:52,637 [243] [DEBUG] [app] Ending request: urn:request:945fc6bf-4e50-4351-a9b3-0440642513ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:945fc6bf-4e50-4351-a9b3-0440642513ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:52,637 [239] [DEBUG] [app] Ending request: urn:request:58039417-7090-408a-bf9c-2295dfa0d534 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58039417-7090-408a-bf9c-2295dfa0d534', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:23:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:52,648 [244] [DEBUG] [app] Ending request: urn:request:c19b6203-e215-4401-928f-3ecdc504e39b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c19b6203-e215-4401-928f-3ecdc504e39b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:52,648 [239] [DEBUG] [app] Ending request: urn:request:a7f25a28-5261-46be-9ed9-51304cf1f51b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7f25a28-5261-46be-9ed9-51304cf1f51b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:23:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:23:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:09.230038+00:00 (in 16.392167 seconds) securityworker stdout | 2023-01-17 05:23:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:22 UTC)" (scheduled at 2023-01-17 05:23:52.837365+00:00) securityworker stdout | 2023-01-17 05:23:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:58.735066+00:00 (in 1.799574 seconds) notificationworker stdout | 2023-01-17 05:23:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:28:56 UTC)" (scheduled at 2023-01-17 05:23:56.935055+00:00) notificationworker stdout | 2023-01-17 05:23:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 23, 56, 935861), 'notification/%']) notificationworker stdout | 2023-01-17 05:23:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 23, 56, 935861), True, datetime.datetime(2023, 1, 17, 5, 23, 56, 935861), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:23:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 23, 56, 935861), True, datetime.datetime(2023, 1, 17, 5, 23, 56, 935861), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 23, 56, 935861), 'notification/%']) notificationworker stdout | 2023-01-17 05:23:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:28:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:23:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:23:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:27.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 05:23:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:27 UTC)" (scheduled at 2023-01-17 05:23:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:23:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 23, 57, 227233), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:23:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:23:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:23:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:08.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 05:23:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:08 UTC)" (scheduled at 2023-01-17 05:23:58.735066+00:00) notificationworker stdout | 2023-01-17 05:23:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 58, 735905), True, datetime.datetime(2023, 1, 17, 5, 23, 58, 735905), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:23:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:23:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:15.931994+00:00 (in 16.594044 seconds) namespacegcworker stdout | 2023-01-17 05:23:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:59 UTC)" (scheduled at 2023-01-17 05:23:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:23:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:23:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 59, 338441), True, datetime.datetime(2023, 1, 17, 5, 23, 59, 338441), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:23:59,355 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:23:59,355 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:23:59,356 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:59 UTC)" executed successfully builder stdout | 2023-01-17 05:23:59,585 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:59,593 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:59,593 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:59,599 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:00,079 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:24:00,231 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:24:00,535 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:24:01,338 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:24:01,366 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:24:02,645 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:24:03,374 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:24:03,396 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:24:03,805 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:24:04,735 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:24:04,751 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:24:05,717 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:24:05,816 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:24:05,818 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:24:05,821 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:24:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:24:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:35.848675+00:00 (in 29.999590 seconds) gcworker stdout | 2023-01-17 05:24:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:35 UTC)" (scheduled at 2023-01-17 05:24:05.848675+00:00) gcworker stdout | 2023-01-17 05:24:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723445849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:24:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:24:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:24:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:35 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:24:06,466 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:24:06,479 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:24:07,626 [239] [DEBUG] [app] Starting request: urn:request:8f87d682-a9f3-4cbf-805a-c3ae90d8ade2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:07,626 [241] [DEBUG] [app] Starting request: urn:request:674c0092-022b-48d0-99e1-cee8f4672c6b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:07,631 [242] [DEBUG] [app] Starting request: urn:request:cae9ac1f-82c8-4cbf-96d6-6a1115a95ea7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:07,631 [240] [DEBUG] [app] Starting request: urn:request:1d80fbff-3f1b-4620-8cbc-b1071493d598 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:07,631 [242] [DEBUG] [app] Ending request: urn:request:cae9ac1f-82c8-4cbf-96d6-6a1115a95ea7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cae9ac1f-82c8-4cbf-96d6-6a1115a95ea7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:24:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:24:07,632 [240] [DEBUG] [app] Ending request: urn:request:1d80fbff-3f1b-4620-8cbc-b1071493d598 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d80fbff-3f1b-4620-8cbc-b1071493d598', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:24:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:24:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:07,636 [244] [DEBUG] [app] Starting request: urn:request:a52fc017-219c-4670-a036-688fe2beef42 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:07,636 [239] [DEBUG] [app] Starting request: urn:request:c8a9d357-5641-47c0-a16f-1ef703bf9eab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:07,636 [244] [DEBUG] [app] Ending request: urn:request:a52fc017-219c-4670-a036-688fe2beef42 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a52fc017-219c-4670-a036-688fe2beef42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:07,637 [239] [DEBUG] [app] Ending request: urn:request:c8a9d357-5641-47c0-a16f-1ef703bf9eab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8a9d357-5641-47c0-a16f-1ef703bf9eab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:24:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:07,648 [239] [DEBUG] [app] Ending request: urn:request:8f87d682-a9f3-4cbf-805a-c3ae90d8ade2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f87d682-a9f3-4cbf-805a-c3ae90d8ade2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:07,648 [241] [DEBUG] [app] Ending request: urn:request:674c0092-022b-48d0-99e1-cee8f4672c6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:674c0092-022b-48d0-99e1-cee8f4672c6b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:24:07,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:24:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:24:07,678 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:24:08,159 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:24:08,161 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:24:08,164 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:24:08,169 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:24:08,171 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:24:08,173 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:24:08,176 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:24:08,179 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:24:08,181 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:24:08,312 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:24:08,386 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:24:08,388 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:24:08,390 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:24:08,394 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:24:08,397 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:24:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:18.735066+00:00 (in 9.999570 seconds) notificationworker stdout | 2023-01-17 05:24:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:18 UTC)" (scheduled at 2023-01-17 05:24:08.735066+00:00) notificationworker stdout | 2023-01-17 05:24:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 8, 735858), True, datetime.datetime(2023, 1, 17, 5, 24, 8, 735858), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:24:08,995 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:24:09,094 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:24:09,131 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:24:09,134 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:24:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:22.837365+00:00 (in 13.606795 seconds) securityworker stdout | 2023-01-17 05:24:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:39 UTC)" (scheduled at 2023-01-17 05:24:09.230038+00:00) securityworker stdout | 2023-01-17 05:24:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:24:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:24:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:24:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:39 UTC)" executed successfully builder stdout | 2023-01-17 05:24:09,609 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:09,617 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:09,618 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:09,623 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:24:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:24:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:47.932490+00:00 (in 31.999958 seconds) namespacegcworker stdout | 2023-01-17 05:24:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:29:15 UTC)" (scheduled at 2023-01-17 05:24:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:24:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 24, 15, 932962), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:24:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 24, 15, 932962), True, datetime.datetime(2023, 1, 17, 5, 24, 15, 932962), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:24:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 24, 15, 932962), True, datetime.datetime(2023, 1, 17, 5, 24, 15, 932962), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 24, 15, 932962), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:24:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:24:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:29:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:28.735066+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 05:24:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:28 UTC)" (scheduled at 2023-01-17 05:24:18.735066+00:00) notificationworker stdout | 2023-01-17 05:24:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 18, 735896), True, datetime.datetime(2023, 1, 17, 5, 24, 18, 735896), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:28 UTC)" executed successfully builder stdout | 2023-01-17 05:24:19,633 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:19,641 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:19,641 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:19,647 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:24:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:24:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:25.334138+00:00 (in 5.091474 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:24:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:20 UTC)" (scheduled at 2023-01-17 05:24:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:24:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:24:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 20, 243055), True, datetime.datetime(2023, 1, 17, 5, 24, 20, 243055), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:24:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:24:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:24:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:24:22,625 [239] [DEBUG] [app] Starting request: urn:request:a18c7c4c-7e0b-4821-8ff4-17ddbe55aefd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:22,625 [243] [DEBUG] [app] Starting request: urn:request:d8a0c8d2-2adf-45fe-b2a7-967e2469a3aa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:22,630 [242] [DEBUG] [app] Starting request: urn:request:bf426b87-750b-4be8-b3e8-eda6328680c5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:22,631 [242] [DEBUG] [app] Ending request: urn:request:bf426b87-750b-4be8-b3e8-eda6328680c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf426b87-750b-4be8-b3e8-eda6328680c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:22,631 [232] [DEBUG] [app] Starting request: urn:request:1a81d8af-d4ae-4deb-aac0-e517e227b364 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:24:22,631 [232] [DEBUG] [app] Ending request: urn:request:1a81d8af-d4ae-4deb-aac0-e517e227b364 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1a81d8af-d4ae-4deb-aac0-e517e227b364', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:24:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:24:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:22,636 [241] [DEBUG] [app] Starting request: urn:request:3f773d65-c434-403c-875e-d322be4b3598 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:22,636 [243] [DEBUG] [app] Starting request: urn:request:e4c1fbdf-7e28-47d9-9b14-c1efa0a05dda (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:22,636 [241] [DEBUG] [app] Ending request: urn:request:3f773d65-c434-403c-875e-d322be4b3598 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f773d65-c434-403c-875e-d322be4b3598', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:22,636 [243] [DEBUG] [app] Ending request: urn:request:e4c1fbdf-7e28-47d9-9b14-c1efa0a05dda (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4c1fbdf-7e28-47d9-9b14-c1efa0a05dda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:22,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:22,644 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:22,649 [239] [DEBUG] [app] Ending request: urn:request:a18c7c4c-7e0b-4821-8ff4-17ddbe55aefd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a18c7c4c-7e0b-4821-8ff4-17ddbe55aefd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:22,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:22,649 [243] [DEBUG] [app] Ending request: urn:request:d8a0c8d2-2adf-45fe-b2a7-967e2469a3aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8a0c8d2-2adf-45fe-b2a7-967e2469a3aa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:22,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:22,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.025) gunicorn-web stdout | 2023-01-17 05:24:22,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:24:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:39.230038+00:00 (in 16.392141 seconds) securityworker stdout | 2023-01-17 05:24:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:52 UTC)" (scheduled at 2023-01-17 05:24:22.837365+00:00) securityworker stdout | 2023-01-17 05:24:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:24:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:24:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:48.830274+00:00 (in 24.188054 seconds) exportactionlogsworker stdout | 2023-01-17 05:24:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:24 UTC)" (scheduled at 2023-01-17 05:24:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:24:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:24:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:24:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:24:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:20.242198+00:00 (in 54.907602 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:24:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:25 UTC)" (scheduled at 2023-01-17 05:24:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:24:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:24:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:24:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:24:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:57.226424+00:00 (in 29.999561 seconds) buildlogsarchiver stdout | 2023-01-17 05:24:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:57 UTC)" (scheduled at 2023-01-17 05:24:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:24:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 24, 27, 227250), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:24:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:24:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:24:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:38.735066+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 05:24:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:38 UTC)" (scheduled at 2023-01-17 05:24:28.735066+00:00) notificationworker stdout | 2023-01-17 05:24:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 28, 735976), True, datetime.datetime(2023, 1, 17, 5, 24, 28, 735976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:38 UTC)" executed successfully builder stdout | 2023-01-17 05:24:29,657 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:29,666 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:29,666 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:29,672 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:30,102 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:24:30,266 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:24:30,560 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:24:31,374 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:24:31,401 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:24:32,661 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:24:33,408 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:24:33,410 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:24:33,839 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:24:34,771 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:24:34,786 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:24:35,727 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:24:35,823 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:24:35,826 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:24:35,828 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:24:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:24:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:05.848675+00:00 (in 29.999611 seconds) gcworker stdout | 2023-01-17 05:24:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:05 UTC)" (scheduled at 2023-01-17 05:24:35.848675+00:00) gcworker stdout | 2023-01-17 05:24:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723475849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:24:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:24:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:24:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:24:36,501 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:24:36,503 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:24:37,625 [243] [DEBUG] [app] Starting request: urn:request:98c1ba7e-00c1-4527-9d0e-51c6e05093ef (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:37,625 [239] [DEBUG] [app] Starting request: urn:request:88aa61d1-4445-4517-9a6b-5ce876ec7d31 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:37,630 [237] [DEBUG] [app] Starting request: urn:request:4a2dcded-14fc-40a1-8a8b-117b3c8f1a9a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:37,631 [236] [DEBUG] [app] Starting request: urn:request:3115cedb-237b-40ed-a891-c77dcd459ae1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:37,631 [237] [DEBUG] [app] Ending request: urn:request:4a2dcded-14fc-40a1-8a8b-117b3c8f1a9a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a2dcded-14fc-40a1-8a8b-117b3c8f1a9a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:24:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:24:37,631 [236] [DEBUG] [app] Ending request: urn:request:3115cedb-237b-40ed-a891-c77dcd459ae1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3115cedb-237b-40ed-a891-c77dcd459ae1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:24:37,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:37,636 [244] [DEBUG] [app] Starting request: urn:request:9428d1a6-4981-4346-b925-4efe3e89e266 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:37,636 [244] [DEBUG] [app] Ending request: urn:request:9428d1a6-4981-4346-b925-4efe3e89e266 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9428d1a6-4981-4346-b925-4efe3e89e266', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:37,636 [243] [DEBUG] [app] Starting request: urn:request:dbc584a0-e612-4b9f-a2f7-81d8bc0f3dc5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:37,637 [243] [DEBUG] [app] Ending request: urn:request:dbc584a0-e612-4b9f-a2f7-81d8bc0f3dc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dbc584a0-e612-4b9f-a2f7-81d8bc0f3dc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:37,647 [243] [DEBUG] [app] Ending request: urn:request:98c1ba7e-00c1-4527-9d0e-51c6e05093ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:98c1ba7e-00c1-4527-9d0e-51c6e05093ef', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:37,648 [239] [DEBUG] [app] Ending request: urn:request:88aa61d1-4445-4517-9a6b-5ce876ec7d31 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:88aa61d1-4445-4517-9a6b-5ce876ec7d31', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:24:37,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:24:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) storagereplication stdout | 2023-01-17 05:24:37,696 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:24:38,166 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:24:38,171 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:24:38,173 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:24:38,177 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:24:38,179 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:24:38,184 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:24:38,186 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:24:38,188 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:24:38,191 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:24:38,338 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:24:38,392 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:24:38,394 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:24:38,397 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:24:38,401 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:24:38,403 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:24:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:43.935629+00:00 (in 5.200146 seconds) notificationworker stdout | 2023-01-17 05:24:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:48 UTC)" (scheduled at 2023-01-17 05:24:38.735066+00:00) notificationworker stdout | 2023-01-17 05:24:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 38, 735874), True, datetime.datetime(2023, 1, 17, 5, 24, 38, 735874), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:24:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:24:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:48.738458+00:00 (in 9.707193 seconds) repositorygcworker stdout | 2023-01-17 05:24:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:39 UTC)" (scheduled at 2023-01-17 05:24:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:24:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. gcworker stdout | 2023-01-17 05:24:39,031 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} repositorygcworker stdout | 2023-01-17 05:24:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 39, 31670), True, datetime.datetime(2023, 1, 17, 5, 24, 39, 31670), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:24:39,047 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:24:39,047 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:24:39,047 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:39 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:24:39,129 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:24:39,148 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:24:39,165 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityworker stdout | 2023-01-17 05:24:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:52.837365+00:00 (in 13.606804 seconds) securityworker stdout | 2023-01-17 05:24:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:09 UTC)" (scheduled at 2023-01-17 05:24:39.230038+00:00) securityworker stdout | 2023-01-17 05:24:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:24:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:24:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:24:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:09 UTC)" executed successfully builder stdout | 2023-01-17 05:24:39,682 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:39,691 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:39,691 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:39,697 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:48.735066+00:00 (in 4.798960 seconds) notificationworker stdout | 2023-01-17 05:24:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:43 UTC)" (scheduled at 2023-01-17 05:24:43.935629+00:00) notificationworker stdout | 2023-01-17 05:24:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:24:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:24:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:24:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:59.337413+00:00 (in 11.404359 seconds) namespacegcworker stdout | 2023-01-17 05:24:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:47 UTC)" (scheduled at 2023-01-17 05:24:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:24:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:24:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:58.735066+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:24:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:58 UTC)" (scheduled at 2023-01-17 05:24:48.735066+00:00) notificationworker stdout | 2023-01-17 05:24:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 48, 735955), True, datetime.datetime(2023, 1, 17, 5, 24, 48, 735955), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:24:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:24:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:39.030814+00:00 (in 50.291914 seconds) repositorygcworker stdout | 2023-01-17 05:24:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:48 UTC)" (scheduled at 2023-01-17 05:24:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:24:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:24:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:24:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:24:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:24.641816+00:00 (in 35.811064 seconds) exportactionlogsworker stdout | 2023-01-17 05:24:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:48 UTC)" (scheduled at 2023-01-17 05:24:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:24:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:24:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 48, 831162), True, datetime.datetime(2023, 1, 17, 5, 24, 48, 831162), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:24:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:24:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:24:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:48 UTC)" executed successfully builder stdout | 2023-01-17 05:24:49,707 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:49,716 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:49,716 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:49,721 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:52,625 [243] [DEBUG] [app] Starting request: urn:request:7f91cfb5-2310-4c29-a594-ade6791ee5e1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:52,625 [239] [DEBUG] [app] Starting request: urn:request:518c38a9-b344-4f2c-ad9c-b97ba872d795 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:24:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:52,630 [240] [DEBUG] [app] Starting request: urn:request:ac4be773-0ec9-421f-96ce-52a2d6949c6a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,630 [232] [DEBUG] [app] Starting request: urn:request:abc70ee7-9110-40da-b396-3296929495c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,631 [240] [DEBUG] [app] Ending request: urn:request:ac4be773-0ec9-421f-96ce-52a2d6949c6a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac4be773-0ec9-421f-96ce-52a2d6949c6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,631 [232] [DEBUG] [app] Ending request: urn:request:abc70ee7-9110-40da-b396-3296929495c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abc70ee7-9110-40da-b396-3296929495c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:24:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:24:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:52,635 [244] [DEBUG] [app] Starting request: urn:request:860e674f-6316-4793-a858-3bf2170889df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:52,635 [243] [DEBUG] [app] Starting request: urn:request:e6353ab3-4218-4370-a6cf-5c7253c49abb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:52,636 [244] [DEBUG] [app] Ending request: urn:request:860e674f-6316-4793-a858-3bf2170889df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:860e674f-6316-4793-a858-3bf2170889df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:52,636 [243] [DEBUG] [app] Ending request: urn:request:e6353ab3-4218-4370-a6cf-5c7253c49abb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6353ab3-4218-4370-a6cf-5c7253c49abb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:52,648 [239] [DEBUG] [app] Ending request: urn:request:518c38a9-b344-4f2c-ad9c-b97ba872d795 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:518c38a9-b344-4f2c-ad9c-b97ba872d795', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:24:52,654 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:52,659 [243] [DEBUG] [app] Ending request: urn:request:7f91cfb5-2310-4c29-a594-ade6791ee5e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f91cfb5-2310-4c29-a594-ade6791ee5e1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:52,659 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:52,660 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) securityworker stdout | 2023-01-17 05:24:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:09.230038+00:00 (in 16.392185 seconds) securityworker stdout | 2023-01-17 05:24:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:22 UTC)" (scheduled at 2023-01-17 05:24:52.837365+00:00) securityworker stdout | 2023-01-17 05:24:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:24:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:24:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:27.226424+00:00 (in 29.999497 seconds) buildlogsarchiver stdout | 2023-01-17 05:24:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:27 UTC)" (scheduled at 2023-01-17 05:24:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:24:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 24, 57, 227340), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:24:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:24:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:24:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:08.735066+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:24:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:08 UTC)" (scheduled at 2023-01-17 05:24:58.735066+00:00) notificationworker stdout | 2023-01-17 05:24:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 58, 736022), True, datetime.datetime(2023, 1, 17, 5, 24, 58, 736022), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:24:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:24:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:47.932490+00:00 (in 48.594563 seconds) namespacegcworker stdout | 2023-01-17 05:24:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:59 UTC)" (scheduled at 2023-01-17 05:24:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:24:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:24:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 59, 338377), True, datetime.datetime(2023, 1, 17, 5, 24, 59, 338377), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:24:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:24:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:24:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:59 UTC)" executed successfully builder stdout | 2023-01-17 05:24:59,731 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:59,740 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:59,740 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:59,746 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:00,137 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:25:00,289 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:25:00,580 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:25:01,385 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:25:01,412 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:25:02,691 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:25:03,415 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:25:03,445 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:25:03,866 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:25:04,787 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:25:04,821 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:25:05,763 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:25:05,831 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:25:05,833 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:25:05,835 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:25:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:25:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:35.848675+00:00 (in 29.999607 seconds) gcworker stdout | 2023-01-17 05:25:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:35 UTC)" (scheduled at 2023-01-17 05:25:05.848675+00:00) gcworker stdout | 2023-01-17 05:25:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723505849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:25:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:25:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:25:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:35 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:25:06,513 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:25:06,530 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:25:07,626 [244] [DEBUG] [app] Starting request: urn:request:c640e74c-1e33-47e1-9389-cc361ed599e5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:07,626 [243] [DEBUG] [app] Starting request: urn:request:444af0e4-dc0a-4dea-9ef7-8a5c60d9e41f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:07,631 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:07,631 [232] [DEBUG] [app] Starting request: urn:request:2a88dc44-f4fb-418b-946d-bffdb73ebf7d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:07,632 [236] [DEBUG] [app] Starting request: urn:request:5bec9494-1a3e-484d-b19e-2934724bd289 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:07,632 [232] [DEBUG] [app] Ending request: urn:request:2a88dc44-f4fb-418b-946d-bffdb73ebf7d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a88dc44-f4fb-418b-946d-bffdb73ebf7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:25:07,632 [236] [DEBUG] [app] Ending request: urn:request:5bec9494-1a3e-484d-b19e-2934724bd289 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5bec9494-1a3e-484d-b19e-2934724bd289', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:25:07,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:07,636 [241] [DEBUG] [app] Starting request: urn:request:7c9bb0a4-f9f1-4ce9-be93-eb5b51cbbc63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:07,637 [239] [DEBUG] [app] Starting request: urn:request:e78e98a2-d77f-4922-8edc-0d2a1d861f37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:07,637 [241] [DEBUG] [app] Ending request: urn:request:7c9bb0a4-f9f1-4ce9-be93-eb5b51cbbc63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c9bb0a4-f9f1-4ce9-be93-eb5b51cbbc63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:25:07,637 [239] [DEBUG] [app] Ending request: urn:request:e78e98a2-d77f-4922-8edc-0d2a1d861f37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e78e98a2-d77f-4922-8edc-0d2a1d861f37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:25:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:07,647 [244] [DEBUG] [app] Ending request: urn:request:c640e74c-1e33-47e1-9389-cc361ed599e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c640e74c-1e33-47e1-9389-cc361ed599e5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:07,648 [243] [DEBUG] [app] Ending request: urn:request:444af0e4-dc0a-4dea-9ef7-8a5c60d9e41f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:444af0e4-dc0a-4dea-9ef7-8a5c60d9e41f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:07,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:25:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:25:07,731 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:25:08,175 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:25:08,179 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:25:08,181 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:25:08,184 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:25:08,187 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:25:08,191 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:25:08,195 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:25:08,198 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:25:08,200 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:25:08,371 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:25:08,398 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:25:08,400 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:25:08,403 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:25:08,407 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:25:08,410 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:25:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:18.735066+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:25:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:18 UTC)" (scheduled at 2023-01-17 05:25:08.735066+00:00) notificationworker stdout | 2023-01-17 05:25:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 8, 735985), True, datetime.datetime(2023, 1, 17, 5, 25, 8, 735985), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:25:09,067 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:25:09,158 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:25:09,160 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:25:09,181 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityworker stdout | 2023-01-17 05:25:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:22.837365+00:00 (in 13.606901 seconds) securityworker stdout | 2023-01-17 05:25:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:39 UTC)" (scheduled at 2023-01-17 05:25:09.230038+00:00) securityworker stdout | 2023-01-17 05:25:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:25:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:25:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:25:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:25:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:25:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:25:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:39 UTC)" executed successfully builder stdout | 2023-01-17 05:25:09,756 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:09,765 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:09,765 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:09,771 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:28.735066+00:00 (in 9.999572 seconds) notificationworker stdout | 2023-01-17 05:25:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:28 UTC)" (scheduled at 2023-01-17 05:25:18.735066+00:00) notificationworker stdout | 2023-01-17 05:25:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 18, 735906), True, datetime.datetime(2023, 1, 17, 5, 25, 18, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:28 UTC)" executed successfully builder stdout | 2023-01-17 05:25:19,781 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:19,790 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:19,790 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:19,796 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:25:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:25:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:25.334138+00:00 (in 5.091522 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:25:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:20 UTC)" (scheduled at 2023-01-17 05:25:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:25:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:25:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 20, 243018), True, datetime.datetime(2023, 1, 17, 5, 25, 20, 243018), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:25:20,257 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:25:20,257 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:25:20,257 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:25:22,626 [243] [DEBUG] [app] Starting request: urn:request:90f258f6-0a9a-4762-955d-be9093c840d1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:22,626 [239] [DEBUG] [app] Starting request: urn:request:4a5f454f-a907-4d3d-b2de-5bda96cb2255 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:22,631 [236] [DEBUG] [app] Starting request: urn:request:4262f1ac-ac8a-4be1-b622-94f13fd7b81c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:22,631 [232] [DEBUG] [app] Starting request: urn:request:3a6575a5-0294-444a-8d2f-d00c669c951a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:22,631 [236] [DEBUG] [app] Ending request: urn:request:4262f1ac-ac8a-4be1-b622-94f13fd7b81c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4262f1ac-ac8a-4be1-b622-94f13fd7b81c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:25:22,631 [232] [DEBUG] [app] Ending request: urn:request:3a6575a5-0294-444a-8d2f-d00c669c951a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a6575a5-0294-444a-8d2f-d00c669c951a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:25:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:22,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:22,637 [244] [DEBUG] [app] Starting request: urn:request:bb07a938-45a4-4f2f-b2b7-f078d5812b32 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:22,637 [239] [DEBUG] [app] Starting request: urn:request:f7a46904-db4c-4cc8-9400-dac110e90f58 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:22,637 [239] [DEBUG] [app] Ending request: urn:request:f7a46904-db4c-4cc8-9400-dac110e90f58 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7a46904-db4c-4cc8-9400-dac110e90f58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:22,637 [244] [DEBUG] [app] Ending request: urn:request:bb07a938-45a4-4f2f-b2b7-f078d5812b32 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb07a938-45a4-4f2f-b2b7-f078d5812b32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:22,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:22,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:22,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:22,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:22,648 [239] [DEBUG] [app] Ending request: urn:request:4a5f454f-a907-4d3d-b2de-5bda96cb2255 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4a5f454f-a907-4d3d-b2de-5bda96cb2255', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:22,648 [243] [DEBUG] [app] Ending request: urn:request:90f258f6-0a9a-4762-955d-be9093c840d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90f258f6-0a9a-4762-955d-be9093c840d1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:25:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:39.230038+00:00 (in 16.392252 seconds) securityworker stdout | 2023-01-17 05:25:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:52 UTC)" (scheduled at 2023-01-17 05:25:22.837365+00:00) securityworker stdout | 2023-01-17 05:25:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:25:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:25:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:48.830274+00:00 (in 24.187981 seconds) exportactionlogsworker stdout | 2023-01-17 05:25:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:24 UTC)" (scheduled at 2023-01-17 05:25:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:25:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:25:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:25:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:25:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:20.242198+00:00 (in 54.907587 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:25:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:25 UTC)" (scheduled at 2023-01-17 05:25:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:25:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:25:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:25:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:25:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:57.226424+00:00 (in 29.999536 seconds) buildlogsarchiver stdout | 2023-01-17 05:25:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:57 UTC)" (scheduled at 2023-01-17 05:25:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:25:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 25, 27, 227267), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:25:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:25:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:25:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:38.735066+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 05:25:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:38 UTC)" (scheduled at 2023-01-17 05:25:28.735066+00:00) notificationworker stdout | 2023-01-17 05:25:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 28, 735933), True, datetime.datetime(2023, 1, 17, 5, 25, 28, 735933), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:38 UTC)" executed successfully builder stdout | 2023-01-17 05:25:29,806 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:29,815 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:29,815 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:29,821 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:30,144 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:25:30,325 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:25:30,588 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:25:31,421 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:25:31,429 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:25:32,713 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:25:33,451 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:25:33,455 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:25:33,898 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:25:34,793 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:25:34,838 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:25:35,786 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:25:35,837 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:25:35,839 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:25:35,841 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:25:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:25:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:05.848675+00:00 (in 29.999590 seconds) gcworker stdout | 2023-01-17 05:25:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:05 UTC)" (scheduled at 2023-01-17 05:25:35.848675+00:00) gcworker stdout | 2023-01-17 05:25:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723535849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:25:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:25:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:25:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:05 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:25:36,542 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:25:36,548 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:25:37,625 [244] [DEBUG] [app] Starting request: urn:request:479eebc0-6d26-45ba-a495-332231773211 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:37,625 [239] [DEBUG] [app] Starting request: urn:request:93a2fb74-a69b-4b9b-b331-775006943ec7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:37,630 [242] [DEBUG] [app] Starting request: urn:request:75f5238b-fe4d-4b14-9fcd-ab4d0110fd8a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:37,630 [232] [DEBUG] [app] Starting request: urn:request:e0bf14ff-5bcc-46d5-8d67-90ef3b8bebf7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:37,631 [242] [DEBUG] [app] Ending request: urn:request:75f5238b-fe4d-4b14-9fcd-ab4d0110fd8a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:75f5238b-fe4d-4b14-9fcd-ab4d0110fd8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:37,631 [232] [DEBUG] [app] Ending request: urn:request:e0bf14ff-5bcc-46d5-8d67-90ef3b8bebf7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e0bf14ff-5bcc-46d5-8d67-90ef3b8bebf7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:25:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:25:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:25:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:37,635 [243] [DEBUG] [app] Starting request: urn:request:6685b4a6-a377-49e1-ae0b-b10c508a2143 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:37,635 [243] [DEBUG] [app] Ending request: urn:request:6685b4a6-a377-49e1-ae0b-b10c508a2143 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6685b4a6-a377-49e1-ae0b-b10c508a2143', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:37,636 [241] [DEBUG] [app] Starting request: urn:request:28cd0f44-6f5c-49a2-9709-b0f66d5b246c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:37,636 [241] [DEBUG] [app] Ending request: urn:request:28cd0f44-6f5c-49a2-9709-b0f66d5b246c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28cd0f44-6f5c-49a2-9709-b0f66d5b246c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:37,646 [239] [DEBUG] [app] Ending request: urn:request:93a2fb74-a69b-4b9b-b331-775006943ec7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93a2fb74-a69b-4b9b-b331-775006943ec7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:37,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:37,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:37,647 [244] [DEBUG] [app] Ending request: urn:request:479eebc0-6d26-45ba-a495-332231773211 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:479eebc0-6d26-45ba-a495-332231773211', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:25:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:25:37,764 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:25:38,184 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:25:38,187 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:25:38,189 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:25:38,192 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:25:38,195 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:25:38,198 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:25:38,202 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:25:38,205 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:25:38,210 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:25:38,376 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:25:38,405 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:25:38,408 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:25:38,410 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:25:38,413 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:25:38,415 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:25:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:43.935629+00:00 (in 5.200156 seconds) notificationworker stdout | 2023-01-17 05:25:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:48 UTC)" (scheduled at 2023-01-17 05:25:38.735066+00:00) notificationworker stdout | 2023-01-17 05:25:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 38, 735856), True, datetime.datetime(2023, 1, 17, 5, 25, 38, 735856), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:25:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:25:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:48.738458+00:00 (in 9.707188 seconds) repositorygcworker stdout | 2023-01-17 05:25:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:39 UTC)" (scheduled at 2023-01-17 05:25:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:25:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:25:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 39, 31722), True, datetime.datetime(2023, 1, 17, 5, 25, 39, 31722), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:25:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:25:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:25:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:25:39,103 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:25:39,169 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:25:39,191 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:25:39,194 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:25:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:52.837365+00:00 (in 13.606909 seconds) securityworker stdout | 2023-01-17 05:25:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:09 UTC)" (scheduled at 2023-01-17 05:25:39.230038+00:00) securityworker stdout | 2023-01-17 05:25:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:25:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:25:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:25:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:25:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:25:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:25:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:09 UTC)" executed successfully builder stdout | 2023-01-17 05:25:39,825 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:39,834 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:39,834 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:39,839 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:48.735066+00:00 (in 4.799006 seconds) notificationworker stdout | 2023-01-17 05:25:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:43 UTC)" (scheduled at 2023-01-17 05:25:43.935629+00:00) notificationworker stdout | 2023-01-17 05:25:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:25:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:25:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:25:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:59.337413+00:00 (in 11.404492 seconds) namespacegcworker stdout | 2023-01-17 05:25:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:47 UTC)" (scheduled at 2023-01-17 05:25:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:25:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:25:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:58.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 05:25:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:58 UTC)" (scheduled at 2023-01-17 05:25:48.735066+00:00) notificationworker stdout | 2023-01-17 05:25:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:48,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 48, 735938), True, datetime.datetime(2023, 1, 17, 5, 25, 48, 735938), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:25:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:25:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:39.030814+00:00 (in 50.291904 seconds) repositorygcworker stdout | 2023-01-17 05:25:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:48 UTC)" (scheduled at 2023-01-17 05:25:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:25:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:25:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:25:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:25:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:24.641816+00:00 (in 35.811066 seconds) exportactionlogsworker stdout | 2023-01-17 05:25:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:48 UTC)" (scheduled at 2023-01-17 05:25:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:25:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:25:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 48, 831204), True, datetime.datetime(2023, 1, 17, 5, 25, 48, 831204), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:25:48,839 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:25:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:25:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:48 UTC)" executed successfully builder stdout | 2023-01-17 05:25:49,850 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:49,859 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:49,859 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:49,864 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:52,626 [244] [DEBUG] [app] Starting request: urn:request:b6eb8a59-ad63-4382-b8b8-8c2583e039ae (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:52,626 [241] [DEBUG] [app] Starting request: urn:request:0a5735c5-02a4-41f2-92ad-47e8a46d5dfe (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:25:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:52,631 [242] [DEBUG] [app] Starting request: urn:request:5feae4b0-c7c2-49d1-80c8-bc2c3c97b3e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,631 [236] [DEBUG] [app] Starting request: urn:request:59ffa31a-a445-4427-8dc1-99d8923dae54 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,632 [242] [DEBUG] [app] Ending request: urn:request:5feae4b0-c7c2-49d1-80c8-bc2c3c97b3e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5feae4b0-c7c2-49d1-80c8-bc2c3c97b3e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,632 [236] [DEBUG] [app] Ending request: urn:request:59ffa31a-a445-4427-8dc1-99d8923dae54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59ffa31a-a445-4427-8dc1-99d8923dae54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:25:52,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:25:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:52,637 [243] [DEBUG] [app] Starting request: urn:request:13ba1d64-ac80-4a4a-a166-d15e94a4aa8d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:52,637 [243] [DEBUG] [app] Ending request: urn:request:13ba1d64-ac80-4a4a-a166-d15e94a4aa8d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:13ba1d64-ac80-4a4a-a166-d15e94a4aa8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:52,637 [241] [DEBUG] [app] Starting request: urn:request:8b2ded53-3fe6-4395-b96e-e4fc2a9accae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:52,638 [241] [DEBUG] [app] Ending request: urn:request:8b2ded53-3fe6-4395-b96e-e4fc2a9accae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b2ded53-3fe6-4395-b96e-e4fc2a9accae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:52,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:52,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:52,648 [244] [DEBUG] [app] Ending request: urn:request:b6eb8a59-ad63-4382-b8b8-8c2583e039ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6eb8a59-ad63-4382-b8b8-8c2583e039ae', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:52,649 [241] [DEBUG] [app] Ending request: urn:request:0a5735c5-02a4-41f2-92ad-47e8a46d5dfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0a5735c5-02a4-41f2-92ad-47e8a46d5dfe', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:52,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:52,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:52,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) securityworker stdout | 2023-01-17 05:25:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:09.230038+00:00 (in 16.392240 seconds) securityworker stdout | 2023-01-17 05:25:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:22 UTC)" (scheduled at 2023-01-17 05:25:52.837365+00:00) securityworker stdout | 2023-01-17 05:25:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:52,859 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:52,861 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:52,861 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:25:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:25:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:27.226424+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 05:25:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:27 UTC)" (scheduled at 2023-01-17 05:25:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:25:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 25, 57, 227266), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:25:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:25:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:25:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:08.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:25:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:08 UTC)" (scheduled at 2023-01-17 05:25:58.735066+00:00) notificationworker stdout | 2023-01-17 05:25:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 58, 735961), True, datetime.datetime(2023, 1, 17, 5, 25, 58, 735961), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:25:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:25:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:47.932490+00:00 (in 48.594570 seconds) namespacegcworker stdout | 2023-01-17 05:25:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:59 UTC)" (scheduled at 2023-01-17 05:25:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:25:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:25:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 59, 338329), True, datetime.datetime(2023, 1, 17, 5, 25, 59, 338329), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:25:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:25:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:25:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:59 UTC)" executed successfully builder stdout | 2023-01-17 05:25:59,875 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:59,883 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:59,883 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:59,889 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:00,150 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:26:00,341 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:26:00,593 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:26:01,455 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:26:01,458 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:26:02,733 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:26:03,487 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:26:03,489 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:26:03,926 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:26:04,828 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:26:04,844 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:26:05,822 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:26:05,843 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:26:05,846 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:26:05,848 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:26:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:26:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:35.848675+00:00 (in 29.999593 seconds) gcworker stdout | 2023-01-17 05:26:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:35 UTC)" (scheduled at 2023-01-17 05:26:05.848675+00:00) gcworker stdout | 2023-01-17 05:26:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723565849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:26:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:26:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:26:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:35 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:26:06,558 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:26:06,564 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:26:07,626 [239] [DEBUG] [app] Starting request: urn:request:5f599b49-d4dd-4288-9912-6b5aa23b6b2b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:07,626 [244] [DEBUG] [app] Starting request: urn:request:2e8361c1-c43c-4384-87d6-d242c229eb87 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:07,631 [232] [DEBUG] [app] Starting request: urn:request:7f8fa2e6-7657-49ab-9d73-232eb1669167 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,631 [240] [DEBUG] [app] Starting request: urn:request:28d88436-6e49-4a88-8074-b115f1070812 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,631 [232] [DEBUG] [app] Ending request: urn:request:7f8fa2e6-7657-49ab-9d73-232eb1669167 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f8fa2e6-7657-49ab-9d73-232eb1669167', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:26:07,632 [240] [DEBUG] [app] Ending request: urn:request:28d88436-6e49-4a88-8074-b115f1070812 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28d88436-6e49-4a88-8074-b115f1070812', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:26:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:26:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:07,636 [243] [DEBUG] [app] Starting request: urn:request:614a6013-7689-4be6-b7ec-ae8f61c600b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:07,636 [241] [DEBUG] [app] Starting request: urn:request:c9f53286-1425-4d2c-ba6f-016f506eaaf2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:07,636 [243] [DEBUG] [app] Ending request: urn:request:614a6013-7689-4be6-b7ec-ae8f61c600b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:614a6013-7689-4be6-b7ec-ae8f61c600b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:07,637 [241] [DEBUG] [app] Ending request: urn:request:c9f53286-1425-4d2c-ba6f-016f506eaaf2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9f53286-1425-4d2c-ba6f-016f506eaaf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:26:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:07,648 [239] [DEBUG] [app] Ending request: urn:request:5f599b49-d4dd-4288-9912-6b5aa23b6b2b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5f599b49-d4dd-4288-9912-6b5aa23b6b2b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:07,648 [244] [DEBUG] [app] Ending request: urn:request:2e8361c1-c43c-4384-87d6-d242c229eb87 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e8361c1-c43c-4384-87d6-d242c229eb87', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:26:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:26:07,780 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:26:08,194 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:26:08,196 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:26:08,199 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:26:08,202 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:26:08,204 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:26:08,206 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:26:08,209 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:26:08,212 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:26:08,219 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:26:08,404 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 05:26:08,411 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:26:08,414 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:26:08,416 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:26:08,418 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:26:08,422 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:26:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:18.735066+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:26:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:18 UTC)" (scheduled at 2023-01-17 05:26:08.735066+00:00) notificationworker stdout | 2023-01-17 05:26:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 8, 735867), True, datetime.datetime(2023, 1, 17, 5, 26, 8, 735867), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:26:09,138 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:26:09,192 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:26:09,201 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:26:09,225 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityworker stdout | 2023-01-17 05:26:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:22.837365+00:00 (in 13.606850 seconds) securityworker stdout | 2023-01-17 05:26:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:39 UTC)" (scheduled at 2023-01-17 05:26:09.230038+00:00) securityworker stdout | 2023-01-17 05:26:09,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:26:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:26:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:26:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:26:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:26:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:26:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:39 UTC)" executed successfully builder stdout | 2023-01-17 05:26:09,895 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:09,904 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:09,904 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:09,910 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:28.735066+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 05:26:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:28 UTC)" (scheduled at 2023-01-17 05:26:18.735066+00:00) notificationworker stdout | 2023-01-17 05:26:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 18, 735909), True, datetime.datetime(2023, 1, 17, 5, 26, 18, 735909), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:28 UTC)" executed successfully builder stdout | 2023-01-17 05:26:19,920 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:19,939 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:19,939 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:19,945 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:26:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:26:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:25.334138+00:00 (in 5.091492 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:26:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:20 UTC)" (scheduled at 2023-01-17 05:26:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:26:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:26:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 20, 243122), True, datetime.datetime(2023, 1, 17, 5, 26, 20, 243122), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:26:20,253 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:26:20,253 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:26:20,253 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:26:22,625 [243] [DEBUG] [app] Starting request: urn:request:cfd3605b-562f-4205-98ed-a59488e2cddf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:22,625 [239] [DEBUG] [app] Starting request: urn:request:75a61454-4363-45c2-9b77-edc89abd100a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:22,630 [240] [DEBUG] [app] Starting request: urn:request:8fb17c2b-8d84-4055-8cd9-05acacf408ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:22,631 [242] [DEBUG] [app] Starting request: urn:request:a6dfe81c-1cbb-4cf3-bd50-8326d203e32b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:22,631 [240] [DEBUG] [app] Ending request: urn:request:8fb17c2b-8d84-4055-8cd9-05acacf408ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fb17c2b-8d84-4055-8cd9-05acacf408ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:26:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:26:22,631 [242] [DEBUG] [app] Ending request: urn:request:a6dfe81c-1cbb-4cf3-bd50-8326d203e32b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a6dfe81c-1cbb-4cf3-bd50-8326d203e32b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:26:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:22,635 [244] [DEBUG] [app] Starting request: urn:request:8c06f369-4709-4140-aff9-76ddb9889f27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:22,636 [244] [DEBUG] [app] Ending request: urn:request:8c06f369-4709-4140-aff9-76ddb9889f27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c06f369-4709-4140-aff9-76ddb9889f27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:22,636 [241] [DEBUG] [app] Starting request: urn:request:bedebac5-f32b-4428-8bfd-07510f23bef6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:26:22,636 [241] [DEBUG] [app] Ending request: urn:request:bedebac5-f32b-4428-8bfd-07510f23bef6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bedebac5-f32b-4428-8bfd-07510f23bef6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:22,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:22,644 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:22,650 [243] [DEBUG] [app] Ending request: urn:request:cfd3605b-562f-4205-98ed-a59488e2cddf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cfd3605b-562f-4205-98ed-a59488e2cddf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:22,650 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:22,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) gunicorn-web stdout | 2023-01-17 05:26:22,651 [239] [DEBUG] [app] Ending request: urn:request:75a61454-4363-45c2-9b77-edc89abd100a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75a61454-4363-45c2-9b77-edc89abd100a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:22,651 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:22,651 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.027) securityworker stdout | 2023-01-17 05:26:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:39.230038+00:00 (in 16.392010 seconds) securityworker stdout | 2023-01-17 05:26:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:52 UTC)" (scheduled at 2023-01-17 05:26:22.837365+00:00) securityworker stdout | 2023-01-17 05:26:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:26:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:26:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:44.641497+00:00 (in 19.999176 seconds) exportactionlogsworker stdout | 2023-01-17 05:26:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:24 UTC)" (scheduled at 2023-01-17 05:26:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:26:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:26:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:26:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:26:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:39.333860+00:00 (in 13.999200 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:26:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:25 UTC)" (scheduled at 2023-01-17 05:26:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:26:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:26:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:26:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:26:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:57.226424+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 05:26:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:57 UTC)" (scheduled at 2023-01-17 05:26:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:26:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 26, 27, 227214), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:26:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:26:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:26:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:38.735066+00:00 (in 9.999494 seconds) notificationworker stdout | 2023-01-17 05:26:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:38 UTC)" (scheduled at 2023-01-17 05:26:28.735066+00:00) notificationworker stdout | 2023-01-17 05:26:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 28, 736005), True, datetime.datetime(2023, 1, 17, 5, 26, 28, 736005), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:38 UTC)" executed successfully builder stdout | 2023-01-17 05:26:29,955 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:29,964 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:29,964 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:29,970 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:30,175 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:26:30,376 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:26:30,608 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:26:31,461 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:26:31,492 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:26:32,766 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:26:33,493 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:26:33,501 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:26:33,940 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:26:34,844 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:26:34,879 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:26:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:26:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:05.848675+00:00 (in 29.999574 seconds) gcworker stdout | 2023-01-17 05:26:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:05 UTC)" (scheduled at 2023-01-17 05:26:35.848675+00:00) chunkcleanupworker stdout | 2023-01-17 05:26:35,849 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gcworker stdout | 2023-01-17 05:26:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723595849, 1209600, True, 3, 500, 1, 0]) gunicorn-secscan stdout | 2023-01-17 05:26:35,852 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:26:35,854 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:26:35,857 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} gcworker stdout | 2023-01-17 05:26:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:26:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:26:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:05 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:26:36,566 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:26:36,590 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:26:37,626 [239] [DEBUG] [app] Starting request: urn:request:a306ea95-182e-4a94-bab1-b616c805caad (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:37,626 [243] [DEBUG] [app] Starting request: urn:request:a041971f-0f27-4f78-8f42-427b497596e1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:37,631 [237] [DEBUG] [app] Starting request: urn:request:c634bec3-f4bb-4b42-a79d-3e5c2cc961e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,631 [242] [DEBUG] [app] Starting request: urn:request:8ed38e50-6df7-47a3-b9ed-663c29484073 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,632 [237] [DEBUG] [app] Ending request: urn:request:c634bec3-f4bb-4b42-a79d-3e5c2cc961e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c634bec3-f4bb-4b42-a79d-3e5c2cc961e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,632 [242] [DEBUG] [app] Ending request: urn:request:8ed38e50-6df7-47a3-b9ed-663c29484073 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ed38e50-6df7-47a3-b9ed-663c29484073', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:26:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:37,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:37,637 [244] [DEBUG] [app] Starting request: urn:request:f4677e97-a52d-4844-8c98-fd9b1b5a489f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:37,637 [244] [DEBUG] [app] Ending request: urn:request:f4677e97-a52d-4844-8c98-fd9b1b5a489f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4677e97-a52d-4844-8c98-fd9b1b5a489f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:37,637 [241] [DEBUG] [app] Starting request: urn:request:d7ff9df8-ba16-4d7b-ab1a-17072a2daa9b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,638 [241] [DEBUG] [app] Ending request: urn:request:d7ff9df8-ba16-4d7b-ab1a-17072a2daa9b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d7ff9df8-ba16-4d7b-ab1a-17072a2daa9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:37,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:37,648 [239] [DEBUG] [app] Ending request: urn:request:a306ea95-182e-4a94-bab1-b616c805caad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a306ea95-182e-4a94-bab1-b616c805caad', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:37,648 [243] [DEBUG] [app] Ending request: urn:request:a041971f-0f27-4f78-8f42-427b497596e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a041971f-0f27-4f78-8f42-427b497596e1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:26:37,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:26:37,816 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:26:38,203 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:26:38,205 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:26:38,207 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:26:38,209 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:26:38,212 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:26:38,214 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:26:38,217 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:26:38,219 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:26:38,226 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:26:38,416 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:26:38,420 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:26:38,422 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:26:38,424 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:26:38,428 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:26:38,437 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:26:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:43.935629+00:00 (in 5.200094 seconds) notificationworker stdout | 2023-01-17 05:26:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:48 UTC)" (scheduled at 2023-01-17 05:26:38.735066+00:00) notificationworker stdout | 2023-01-17 05:26:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 38, 735965), True, datetime.datetime(2023, 1, 17, 5, 26, 38, 735965), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:26:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:26:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:48.738458+00:00 (in 9.707180 seconds) repositorygcworker stdout | 2023-01-17 05:26:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:39 UTC)" (scheduled at 2023-01-17 05:26:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:26:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:26:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 39, 31781), True, datetime.datetime(2023, 1, 17, 5, 26, 39, 31781), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:26:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:26:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:26:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:26:39,165 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:26:39,226 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityworker stdout | 2023-01-17 05:26:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:52.837365+00:00 (in 13.606906 seconds) securityworker stdout | 2023-01-17 05:26:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:09 UTC)" (scheduled at 2023-01-17 05:26:39.230038+00:00) securityworker stdout | 2023-01-17 05:26:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:26:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:26:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityscanningnotificationworker stdout | 2023-01-17 05:26:39,236 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:26:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:09 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:26:39,259 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:26:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:26:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:20.242198+00:00 (in 40.907937 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:26:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:39 UTC)" (scheduled at 2023-01-17 05:26:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:26:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 26, 39, 334628), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:26:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 39, 334628), True, datetime.datetime(2023, 1, 17, 5, 26, 39, 334628), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:26:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 39, 334628), True, datetime.datetime(2023, 1, 17, 5, 26, 39, 334628), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 26, 39, 334628), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:26:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:26:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:39 UTC)" executed successfully builder stdout | 2023-01-17 05:26:39,977 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:39,987 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:39,987 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:39,993 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:48.735066+00:00 (in 4.798951 seconds) notificationworker stdout | 2023-01-17 05:26:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:43 UTC)" (scheduled at 2023-01-17 05:26:43.935629+00:00) notificationworker stdout | 2023-01-17 05:26:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:26:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:26:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:26:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:48.830274+00:00 (in 4.188330 seconds) exportactionlogsworker stdout | 2023-01-17 05:26:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:44 UTC)" (scheduled at 2023-01-17 05:26:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:26:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 26, 44, 642324), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:26:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 44, 642324), True, datetime.datetime(2023, 1, 17, 5, 26, 44, 642324), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:26:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 44, 642324), True, datetime.datetime(2023, 1, 17, 5, 26, 44, 642324), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 26, 44, 642324), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:26:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:26:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:26:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:26:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:59.337413+00:00 (in 11.404445 seconds) namespacegcworker stdout | 2023-01-17 05:26:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:47 UTC)" (scheduled at 2023-01-17 05:26:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:26:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:26:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:58.735066+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 05:26:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:58 UTC)" (scheduled at 2023-01-17 05:26:48.735066+00:00) notificationworker stdout | 2023-01-17 05:26:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 48, 735960), True, datetime.datetime(2023, 1, 17, 5, 26, 48, 735960), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:26:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:26:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:04.738062+00:00 (in 15.999079 seconds) repositorygcworker stdout | 2023-01-17 05:26:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:48 UTC)" (scheduled at 2023-01-17 05:26:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:26:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:26:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:26:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:26:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:24.641816+00:00 (in 35.811134 seconds) exportactionlogsworker stdout | 2023-01-17 05:26:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:48 UTC)" (scheduled at 2023-01-17 05:26:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:26:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:26:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 48, 831065), True, datetime.datetime(2023, 1, 17, 5, 26, 48, 831065), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:26:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:26:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:26:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:48 UTC)" executed successfully builder stdout | 2023-01-17 05:26:50,003 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:50,012 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:50,012 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:50,017 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:52,625 [239] [DEBUG] [app] Starting request: urn:request:bba6162e-6b55-403c-bdc0-fdc392f25c7d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:52,625 [244] [DEBUG] [app] Starting request: urn:request:a28be817-5d3f-4b12-a8bc-7b0f456edeea (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:26:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:52,630 [240] [DEBUG] [app] Starting request: urn:request:dc4437c0-c463-460c-b4d2-603fdbb15546 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,631 [232] [DEBUG] [app] Starting request: urn:request:dfcb4361-aa79-4a5c-803d-017d693c8825 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,631 [240] [DEBUG] [app] Ending request: urn:request:dc4437c0-c463-460c-b4d2-603fdbb15546 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc4437c0-c463-460c-b4d2-603fdbb15546', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:26:52,631 [232] [DEBUG] [app] Ending request: urn:request:dfcb4361-aa79-4a5c-803d-017d693c8825 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dfcb4361-aa79-4a5c-803d-017d693c8825', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:26:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:26:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:52,635 [243] [DEBUG] [app] Starting request: urn:request:3427221c-d1d3-4bf7-9f59-0813a5bd834e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:52,636 [243] [DEBUG] [app] Ending request: urn:request:3427221c-d1d3-4bf7-9f59-0813a5bd834e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3427221c-d1d3-4bf7-9f59-0813a5bd834e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:52,636 [241] [DEBUG] [app] Starting request: urn:request:ad6355e5-ac95-42e0-9f99-a44f6e086d87 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:26:52,636 [241] [DEBUG] [app] Ending request: urn:request:ad6355e5-ac95-42e0-9f99-a44f6e086d87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad6355e5-ac95-42e0-9f99-a44f6e086d87', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:52,647 [244] [DEBUG] [app] Ending request: urn:request:a28be817-5d3f-4b12-a8bc-7b0f456edeea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a28be817-5d3f-4b12-a8bc-7b0f456edeea', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:52,647 [239] [DEBUG] [app] Ending request: urn:request:bba6162e-6b55-403c-bdc0-fdc392f25c7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bba6162e-6b55-403c-bdc0-fdc392f25c7d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:26:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:26:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:09.230038+00:00 (in 16.392260 seconds) securityworker stdout | 2023-01-17 05:26:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:22 UTC)" (scheduled at 2023-01-17 05:26:52.837365+00:00) securityworker stdout | 2023-01-17 05:26:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:26:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:26:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:27.226424+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 05:26:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:27 UTC)" (scheduled at 2023-01-17 05:26:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:26:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 26, 57, 227264), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:26:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:26:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:26:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:08.735066+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 05:26:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:08 UTC)" (scheduled at 2023-01-17 05:26:58.735066+00:00) notificationworker stdout | 2023-01-17 05:26:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 58, 735985), True, datetime.datetime(2023, 1, 17, 5, 26, 58, 735985), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:26:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:26:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:47.932490+00:00 (in 48.594643 seconds) namespacegcworker stdout | 2023-01-17 05:26:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:59 UTC)" (scheduled at 2023-01-17 05:26:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:26:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:26:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 59, 338231), True, datetime.datetime(2023, 1, 17, 5, 26, 59, 338231), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:26:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:26:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:26:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:59 UTC)" executed successfully builder stdout | 2023-01-17 05:27:00,026 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:00,034 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:00,035 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:00,040 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:00,208 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:27:00,400 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:27:00,631 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:27:01,497 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:27:01,519 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:27:02,800 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:27:03,510 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:27:03,528 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:27:03,976 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositorygcworker stdout | 2023-01-17 05:27:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:27:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:39.030814+00:00 (in 34.292306 seconds) repositorygcworker stdout | 2023-01-17 05:27:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:32:04 UTC)" (scheduled at 2023-01-17 05:27:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:27:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 27, 4, 738902), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:27:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 27, 4, 738902), True, datetime.datetime(2023, 1, 17, 5, 27, 4, 738902), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:27:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 27, 4, 738902), True, datetime.datetime(2023, 1, 17, 5, 27, 4, 738902), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 27, 4, 738902), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:27:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:27:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:32:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:27:04,858 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:27:04,913 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:27:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:27:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:35.848675+00:00 (in 29.999583 seconds) gcworker stdout | 2023-01-17 05:27:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:35 UTC)" (scheduled at 2023-01-17 05:27:05.848675+00:00) gcworker stdout | 2023-01-17 05:27:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723625849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:27:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:27:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:27:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:27:05,860 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:27:05,862 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:27:05,864 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:27:05,867 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:27:06,581 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:27:06,620 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:27:07,625 [239] [DEBUG] [app] Starting request: urn:request:e93c7bf3-1b85-4dfc-ac22-b18e42a41227 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:07,625 [243] [DEBUG] [app] Starting request: urn:request:04595caf-45c1-4eda-8b64-048a5449e4e9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:07,629 [232] [DEBUG] [app] Starting request: urn:request:f321af92-b753-4690-bf44-733799bbc8c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,630 [232] [DEBUG] [app] Ending request: urn:request:f321af92-b753-4690-bf44-733799bbc8c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f321af92-b753-4690-bf44-733799bbc8c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,630 [242] [DEBUG] [app] Starting request: urn:request:bd809c88-1d4e-4fd2-943b-7f52cee75793 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:27:07,630 [242] [DEBUG] [app] Ending request: urn:request:bd809c88-1d4e-4fd2-943b-7f52cee75793 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd809c88-1d4e-4fd2-943b-7f52cee75793', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:07,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:27:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:27:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,633 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:07,635 [239] [DEBUG] [app] Starting request: urn:request:28f773d4-5429-45ad-94b6-8a6804fd1297 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:07,634 [244] [DEBUG] [app] Starting request: urn:request:98b38dae-c6b9-4dad-b9f8-cd86cf305d46 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:07,635 [239] [DEBUG] [app] Ending request: urn:request:28f773d4-5429-45ad-94b6-8a6804fd1297 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28f773d4-5429-45ad-94b6-8a6804fd1297', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:07,635 [244] [DEBUG] [app] Ending request: urn:request:98b38dae-c6b9-4dad-b9f8-cd86cf305d46 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98b38dae-c6b9-4dad-b9f8-cd86cf305d46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:07,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:07,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:27:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:07,646 [239] [DEBUG] [app] Ending request: urn:request:e93c7bf3-1b85-4dfc-ac22-b18e42a41227 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e93c7bf3-1b85-4dfc-ac22-b18e42a41227', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:07,646 [243] [DEBUG] [app] Ending request: urn:request:04595caf-45c1-4eda-8b64-048a5449e4e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04595caf-45c1-4eda-8b64-048a5449e4e9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:07,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:07,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:27:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:27:07,851 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:27:08,210 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:27:08,213 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:27:08,215 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:27:08,217 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:27:08,220 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:27:08,223 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:27:08,225 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:27:08,227 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:27:08,234 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:27:08,422 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:27:08,426 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:27:08,429 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:27:08,431 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:27:08,434 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:27:08,450 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:27:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:18.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 05:27:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:18 UTC)" (scheduled at 2023-01-17 05:27:08.735066+00:00) notificationworker stdout | 2023-01-17 05:27:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 8, 735916), True, datetime.datetime(2023, 1, 17, 5, 27, 8, 735916), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:18 UTC)" executed successfully gcworker stdout | 2023-01-17 05:27:09,201 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityworker stdout | 2023-01-17 05:27:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:22.837365+00:00 (in 13.606892 seconds) securityworker stdout | 2023-01-17 05:27:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:39 UTC)" (scheduled at 2023-01-17 05:27:09.230038+00:00) securityworker stdout | 2023-01-17 05:27:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:27:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:27:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:27:09,242 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} securityworker stdout | 2023-01-17 05:27:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:27:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:39 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:27:09,259 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:27:09,282 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:27:10,050 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:10,060 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:10,060 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:10,066 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:28.735066+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:27:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:28 UTC)" (scheduled at 2023-01-17 05:27:18.735066+00:00) notificationworker stdout | 2023-01-17 05:27:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 18, 735914), True, datetime.datetime(2023, 1, 17, 5, 27, 18, 735914), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:28 UTC)" executed successfully builder stdout | 2023-01-17 05:27:20,076 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:20,085 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:20,085 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:20,091 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:27:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:27:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:25.334138+00:00 (in 5.091492 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:27:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:20 UTC)" (scheduled at 2023-01-17 05:27:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:27:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:27:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 20, 243034), True, datetime.datetime(2023, 1, 17, 5, 27, 20, 243034), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:27:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:27:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:27:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:27:22,625 [243] [DEBUG] [app] Starting request: urn:request:e3e72bc6-c32d-442b-872b-b7e8bef6b4fd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:22,626 [239] [DEBUG] [app] Starting request: urn:request:daecdd2a-ca68-48b4-bc39-d5f64daa946f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:22,631 [232] [DEBUG] [app] Starting request: urn:request:b54bc836-8cea-4447-b89b-47eece403863 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,631 [232] [DEBUG] [app] Ending request: urn:request:b54bc836-8cea-4447-b89b-47eece403863 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b54bc836-8cea-4447-b89b-47eece403863', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,631 [236] [DEBUG] [app] Starting request: urn:request:af60e9b0-163d-4a35-a7ee-1edbc275e3a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:27:22,631 [236] [DEBUG] [app] Ending request: urn:request:af60e9b0-163d-4a35-a7ee-1edbc275e3a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af60e9b0-163d-4a35-a7ee-1edbc275e3a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:27:22,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:27:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:22,636 [244] [DEBUG] [app] Starting request: urn:request:0163507d-0d25-4c4e-95af-ec547f2e31ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:22,636 [241] [DEBUG] [app] Starting request: urn:request:fca72d98-ce3d-4b45-b5bc-7bf75c2758d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:22,636 [244] [DEBUG] [app] Ending request: urn:request:0163507d-0d25-4c4e-95af-ec547f2e31ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0163507d-0d25-4c4e-95af-ec547f2e31ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:22,636 [241] [DEBUG] [app] Ending request: urn:request:fca72d98-ce3d-4b45-b5bc-7bf75c2758d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fca72d98-ce3d-4b45-b5bc-7bf75c2758d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:22,647 [239] [DEBUG] [app] Ending request: urn:request:daecdd2a-ca68-48b4-bc39-d5f64daa946f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:daecdd2a-ca68-48b4-bc39-d5f64daa946f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:22,647 [243] [DEBUG] [app] Ending request: urn:request:e3e72bc6-c32d-442b-872b-b7e8bef6b4fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3e72bc6-c32d-442b-872b-b7e8bef6b4fd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:27:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:27:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:39.230038+00:00 (in 16.392269 seconds) securityworker stdout | 2023-01-17 05:27:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:52 UTC)" (scheduled at 2023-01-17 05:27:22.837365+00:00) securityworker stdout | 2023-01-17 05:27:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:22,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:27:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:27:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:48.830274+00:00 (in 24.187994 seconds) exportactionlogsworker stdout | 2023-01-17 05:27:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:24 UTC)" (scheduled at 2023-01-17 05:27:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:27:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:27:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:27:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:27:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:20.242198+00:00 (in 54.907600 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:27:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:25 UTC)" (scheduled at 2023-01-17 05:27:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:27:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:27:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:27:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:27:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:57.226424+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:27:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:57 UTC)" (scheduled at 2023-01-17 05:27:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:27:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 27, 27, 227282), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:27:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:27:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:27:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:38.735066+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:27:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:38 UTC)" (scheduled at 2023-01-17 05:27:28.735066+00:00) notificationworker stdout | 2023-01-17 05:27:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 28, 735985), True, datetime.datetime(2023, 1, 17, 5, 27, 28, 735985), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:28,746 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:28,746 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:28,746 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:38 UTC)" executed successfully builder stdout | 2023-01-17 05:27:30,101 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:30,110 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:30,110 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:30,116 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:30,243 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:27:30,430 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:27:30,662 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:27:31,533 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:27:31,542 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:27:32,836 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:27:33,545 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:27:33,563 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:27:34,012 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:27:34,877 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:27:34,948 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:27:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:27:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:05.848675+00:00 (in 29.999533 seconds) gcworker stdout | 2023-01-17 05:27:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:05 UTC)" (scheduled at 2023-01-17 05:27:35.848675+00:00) gcworker stdout | 2023-01-17 05:27:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723655849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:27:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:27:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:27:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:05 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:27:35,871 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:27:35,874 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:27:35,877 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:27:35,879 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} queuecleanupworker stdout | 2023-01-17 05:27:36,595 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:27:36,656 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:27:37,625 [239] [DEBUG] [app] Starting request: urn:request:e6ebba02-32a0-491d-87d4-455d8a508add (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:37,625 [244] [DEBUG] [app] Starting request: urn:request:4618747a-a8f5-402d-a37f-21ef4d202863 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:37,630 [232] [DEBUG] [app] Starting request: urn:request:4f3a8241-31e5-4c2d-956f-be730b3186ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,630 [240] [DEBUG] [app] Starting request: urn:request:2f8edd34-ed90-43ca-a4ac-c5a32cf9f9bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,630 [232] [DEBUG] [app] Ending request: urn:request:4f3a8241-31e5-4c2d-956f-be730b3186ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f3a8241-31e5-4c2d-956f-be730b3186ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,631 [240] [DEBUG] [app] Ending request: urn:request:2f8edd34-ed90-43ca-a4ac-c5a32cf9f9bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f8edd34-ed90-43ca-a4ac-c5a32cf9f9bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:27:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:27:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:37,635 [243] [DEBUG] [app] Starting request: urn:request:ebc2ff3b-b05a-4f5e-8eb1-5d162403330f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:37,635 [241] [DEBUG] [app] Starting request: urn:request:957ee506-24e5-4c47-8625-6639ae0027f0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:37,635 [243] [DEBUG] [app] Ending request: urn:request:ebc2ff3b-b05a-4f5e-8eb1-5d162403330f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ebc2ff3b-b05a-4f5e-8eb1-5d162403330f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:37,636 [241] [DEBUG] [app] Ending request: urn:request:957ee506-24e5-4c47-8625-6639ae0027f0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:957ee506-24e5-4c47-8625-6639ae0027f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:27:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:37,647 [244] [DEBUG] [app] Ending request: urn:request:4618747a-a8f5-402d-a37f-21ef4d202863 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4618747a-a8f5-402d-a37f-21ef4d202863', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:37,647 [239] [DEBUG] [app] Ending request: urn:request:e6ebba02-32a0-491d-87d4-455d8a508add (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6ebba02-32a0-491d-87d4-455d8a508add', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:27:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:27:37,886 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:27:38,217 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:27:38,220 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:27:38,223 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:27:38,226 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:27:38,229 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:27:38,231 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:27:38,234 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:27:38,236 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:27:38,241 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:27:38,428 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:27:38,432 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:27:38,434 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:27:38,436 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:27:38,440 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:27:38,482 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:27:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:43.935629+00:00 (in 5.200095 seconds) notificationworker stdout | 2023-01-17 05:27:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:48 UTC)" (scheduled at 2023-01-17 05:27:38.735066+00:00) notificationworker stdout | 2023-01-17 05:27:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 38, 735949), True, datetime.datetime(2023, 1, 17, 5, 27, 38, 735949), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:27:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:27:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:48.738458+00:00 (in 9.707186 seconds) repositorygcworker stdout | 2023-01-17 05:27:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:39 UTC)" (scheduled at 2023-01-17 05:27:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:27:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:27:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 39, 31681), True, datetime.datetime(2023, 1, 17, 5, 27, 39, 31681), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:27:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:27:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:27:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:27:39,208 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityworker stdout | 2023-01-17 05:27:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:52.837365+00:00 (in 13.606887 seconds) securityworker stdout | 2023-01-17 05:27:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:09 UTC)" (scheduled at 2023-01-17 05:27:39.230038+00:00) securityworker stdout | 2023-01-17 05:27:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:27:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:27:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:27:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:27:39,277 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:27:39,286 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:27:39,307 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:27:40,126 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:40,135 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:40,135 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:40,142 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:48.735066+00:00 (in 4.798977 seconds) notificationworker stdout | 2023-01-17 05:27:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:43 UTC)" (scheduled at 2023-01-17 05:27:43.935629+00:00) notificationworker stdout | 2023-01-17 05:27:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:27:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:27:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:27:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:59.337413+00:00 (in 11.404472 seconds) namespacegcworker stdout | 2023-01-17 05:27:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:47 UTC)" (scheduled at 2023-01-17 05:27:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:27:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:27:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:58.735066+00:00 (in 9.999559 seconds) notificationworker stdout | 2023-01-17 05:27:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:58 UTC)" (scheduled at 2023-01-17 05:27:48.735066+00:00) notificationworker stdout | 2023-01-17 05:27:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 48, 735917), True, datetime.datetime(2023, 1, 17, 5, 27, 48, 735917), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:27:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:27:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:39.030814+00:00 (in 50.291925 seconds) repositorygcworker stdout | 2023-01-17 05:27:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:48 UTC)" (scheduled at 2023-01-17 05:27:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:27:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:27:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:27:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:27:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:24.641816+00:00 (in 35.811071 seconds) exportactionlogsworker stdout | 2023-01-17 05:27:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:48 UTC)" (scheduled at 2023-01-17 05:27:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:27:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:27:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 48, 831162), True, datetime.datetime(2023, 1, 17, 5, 27, 48, 831162), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:27:48,839 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:27:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:27:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:48 UTC)" executed successfully builder stdout | 2023-01-17 05:27:50,152 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:50,161 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:50,161 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:50,167 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:52,625 [239] [DEBUG] [app] Starting request: urn:request:cca65f96-3f6f-4116-b85b-56eb57310c95 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:52,626 [243] [DEBUG] [app] Starting request: urn:request:959ddc3c-8701-4af2-9c8f-4037fccc9370 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:27:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:52,630 [242] [DEBUG] [app] Starting request: urn:request:ce82c8b4-6528-4b5f-81ee-897afd56e900 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:52,631 [232] [DEBUG] [app] Starting request: urn:request:e6cc786e-4275-4ad4-8a3d-15213c2d090b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:52,631 [242] [DEBUG] [app] Ending request: urn:request:ce82c8b4-6528-4b5f-81ee-897afd56e900 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ce82c8b4-6528-4b5f-81ee-897afd56e900', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:27:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:27:52,631 [232] [DEBUG] [app] Ending request: urn:request:e6cc786e-4275-4ad4-8a3d-15213c2d090b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e6cc786e-4275-4ad4-8a3d-15213c2d090b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:27:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:27:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:52,635 [244] [DEBUG] [app] Starting request: urn:request:7353145e-6722-4eb6-9f3f-676cbe263589 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:52,636 [239] [DEBUG] [app] Starting request: urn:request:bed8361f-8850-4588-9545-632c886c95b2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:52,636 [244] [DEBUG] [app] Ending request: urn:request:7353145e-6722-4eb6-9f3f-676cbe263589 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7353145e-6722-4eb6-9f3f-676cbe263589', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:52,636 [239] [DEBUG] [app] Ending request: urn:request:bed8361f-8850-4588-9545-632c886c95b2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bed8361f-8850-4588-9545-632c886c95b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:52,647 [239] [DEBUG] [app] Ending request: urn:request:cca65f96-3f6f-4116-b85b-56eb57310c95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cca65f96-3f6f-4116-b85b-56eb57310c95', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:52,647 [243] [DEBUG] [app] Ending request: urn:request:959ddc3c-8701-4af2-9c8f-4037fccc9370 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:959ddc3c-8701-4af2-9c8f-4037fccc9370', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:27:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:27:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:09.230038+00:00 (in 16.392209 seconds) securityworker stdout | 2023-01-17 05:27:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:22 UTC)" (scheduled at 2023-01-17 05:27:52.837365+00:00) securityworker stdout | 2023-01-17 05:27:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:27:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:27:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:27.226424+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 05:27:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:27 UTC)" (scheduled at 2023-01-17 05:27:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:27:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 27, 57, 227304), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:27:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:27:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:27:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:08.735066+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:27:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:08 UTC)" (scheduled at 2023-01-17 05:27:58.735066+00:00) notificationworker stdout | 2023-01-17 05:27:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 58, 735950), True, datetime.datetime(2023, 1, 17, 5, 27, 58, 735950), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:27:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:27:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:47.932490+00:00 (in 48.594670 seconds) namespacegcworker stdout | 2023-01-17 05:27:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:59 UTC)" (scheduled at 2023-01-17 05:27:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:27:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:27:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 59, 338247), True, datetime.datetime(2023, 1, 17, 5, 27, 59, 338247), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:27:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:27:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:27:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:59 UTC)" executed successfully builder stdout | 2023-01-17 05:28:00,177 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:00,186 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:00,186 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:00,192 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:00,279 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:28:00,461 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:28:00,697 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:28:01,557 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:28:01,577 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:28:02,843 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:28:03,581 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:28:03,584 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:28:04,047 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:28:04,901 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:28:04,983 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:28:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:28:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:35.848675+00:00 (in 29.999520 seconds) gcworker stdout | 2023-01-17 05:28:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:35 UTC)" (scheduled at 2023-01-17 05:28:05.848675+00:00) gcworker stdout | 2023-01-17 05:28:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723685849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:28:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:28:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:28:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:28:05,881 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:28:05,884 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:28:05,887 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:28:05,904 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:28:06,630 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:28:06,662 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:28:07,625 [239] [DEBUG] [app] Starting request: urn:request:53075b2c-c8de-49d3-b83d-7a671688fdaa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:07,625 [244] [DEBUG] [app] Starting request: urn:request:6c24a920-0f35-4848-909f-fdcbbe60344e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:07,630 [232] [DEBUG] [app] Starting request: urn:request:13682116-3b1e-4af3-94d9-99b1647210dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:07,630 [232] [DEBUG] [app] Ending request: urn:request:13682116-3b1e-4af3-94d9-99b1647210dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13682116-3b1e-4af3-94d9-99b1647210dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:07,630 [240] [DEBUG] [app] Starting request: urn:request:b8ef848a-c6cb-4905-bc03-f4e428b80e7a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:28:07,631 [240] [DEBUG] [app] Ending request: urn:request:b8ef848a-c6cb-4905-bc03-f4e428b80e7a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b8ef848a-c6cb-4905-bc03-f4e428b80e7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:28:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:07,635 [243] [DEBUG] [app] Starting request: urn:request:b9aa5cda-0684-450e-aa4e-ad743f47b49f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:07,635 [241] [DEBUG] [app] Starting request: urn:request:26dd77b3-11d2-45e6-b6be-7a1ff76a795c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:07,636 [243] [DEBUG] [app] Ending request: urn:request:b9aa5cda-0684-450e-aa4e-ad743f47b49f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9aa5cda-0684-450e-aa4e-ad743f47b49f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:07,636 [241] [DEBUG] [app] Ending request: urn:request:26dd77b3-11d2-45e6-b6be-7a1ff76a795c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26dd77b3-11d2-45e6-b6be-7a1ff76a795c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:28:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:07,647 [239] [DEBUG] [app] Ending request: urn:request:53075b2c-c8de-49d3-b83d-7a671688fdaa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:53075b2c-c8de-49d3-b83d-7a671688fdaa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:07,647 [244] [DEBUG] [app] Ending request: urn:request:6c24a920-0f35-4848-909f-fdcbbe60344e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c24a920-0f35-4848-909f-fdcbbe60344e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:28:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:28:07,915 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:28:08,226 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:28:08,228 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:28:08,231 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:28:08,233 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:28:08,236 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:28:08,239 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:28:08,242 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:28:08,245 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:28:08,248 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:28:08,434 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:28:08,437 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:28:08,439 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:28:08,442 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:28:08,447 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:28:08,516 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:28:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:18.735066+00:00 (in 9.999601 seconds) notificationworker stdout | 2023-01-17 05:28:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:18 UTC)" (scheduled at 2023-01-17 05:28:08.735066+00:00) notificationworker stdout | 2023-01-17 05:28:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 8, 735863), True, datetime.datetime(2023, 1, 17, 5, 28, 8, 735863), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:28:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:22.837365+00:00 (in 13.606893 seconds) securityworker stdout | 2023-01-17 05:28:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:39 UTC)" (scheduled at 2023-01-17 05:28:09.230038+00:00) securityworker stdout | 2023-01-17 05:28:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:28:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:28:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) gcworker stdout | 2023-01-17 05:28:09,239 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityworker stdout | 2023-01-17 05:28:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:28:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:28:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:28:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:28:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:39 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:28:09,285 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:28:09,320 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:28:09,335 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:28:10,202 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:10,211 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:10,211 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:10,217 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:28.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:28:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:28 UTC)" (scheduled at 2023-01-17 05:28:18.735066+00:00) notificationworker stdout | 2023-01-17 05:28:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 18, 735944), True, datetime.datetime(2023, 1, 17, 5, 28, 18, 735944), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:28 UTC)" executed successfully builder stdout | 2023-01-17 05:28:20,227 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:20,237 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:20,237 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics securityscanningnotificationworker stdout | 2023-01-17 05:28:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:28:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:25.334138+00:00 (in 5.091470 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:28:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:20 UTC)" (scheduled at 2023-01-17 05:28:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:28:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. builder stdout | 2023-01-17 05:28:20,243 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:28:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 20, 243049), True, datetime.datetime(2023, 1, 17, 5, 28, 20, 243049), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:28:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:28:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:28:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:28:22,626 [244] [DEBUG] [app] Starting request: urn:request:17558054-f1c0-4d0e-95eb-d9f4a2b6b287 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:22,626 [239] [DEBUG] [app] Starting request: urn:request:40a4d624-d97e-4fcd-9865-05e9f6169254 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:22,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:22,632 [242] [DEBUG] [app] Starting request: urn:request:796faef2-0707-4037-ad6a-0492ee5083d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,632 [232] [DEBUG] [app] Starting request: urn:request:768883a5-7c23-4361-8985-d4677a22d437 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,632 [242] [DEBUG] [app] Ending request: urn:request:796faef2-0707-4037-ad6a-0492ee5083d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:796faef2-0707-4037-ad6a-0492ee5083d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:28:22,632 [232] [DEBUG] [app] Ending request: urn:request:768883a5-7c23-4361-8985-d4677a22d437 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:768883a5-7c23-4361-8985-d4677a22d437', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:28:22,633 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:28:22,633 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:22,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:22,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:22,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:22,637 [243] [DEBUG] [app] Starting request: urn:request:83019f14-5d62-478b-adf4-c150fab7f9d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:22,638 [244] [DEBUG] [app] Starting request: urn:request:df384892-281f-45c0-b872-0540f059c7c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:22,638 [243] [DEBUG] [app] Ending request: urn:request:83019f14-5d62-478b-adf4-c150fab7f9d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83019f14-5d62-478b-adf4-c150fab7f9d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:22,638 [244] [DEBUG] [app] Ending request: urn:request:df384892-281f-45c0-b872-0540f059c7c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df384892-281f-45c0-b872-0540f059c7c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:28:22,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:22,639 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:22,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:22,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:22,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:22,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:22,649 [239] [DEBUG] [app] Ending request: urn:request:40a4d624-d97e-4fcd-9865-05e9f6169254 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40a4d624-d97e-4fcd-9865-05e9f6169254', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:22,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:22,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 05:28:22,660 [244] [DEBUG] [app] Ending request: urn:request:17558054-f1c0-4d0e-95eb-d9f4a2b6b287 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:17558054-f1c0-4d0e-95eb-d9f4a2b6b287', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:22,660 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:22,660 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) securityworker stdout | 2023-01-17 05:28:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:39.230038+00:00 (in 16.392241 seconds) securityworker stdout | 2023-01-17 05:28:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:52 UTC)" (scheduled at 2023-01-17 05:28:22.837365+00:00) securityworker stdout | 2023-01-17 05:28:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:28:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:28:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:48.830274+00:00 (in 24.187975 seconds) exportactionlogsworker stdout | 2023-01-17 05:28:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:24 UTC)" (scheduled at 2023-01-17 05:28:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:28:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:28:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:28:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:28:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:20.242198+00:00 (in 54.907628 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:28:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:25 UTC)" (scheduled at 2023-01-17 05:28:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:28:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:28:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:28:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:28:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:57.226424+00:00 (in 29.999541 seconds) buildlogsarchiver stdout | 2023-01-17 05:28:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:57 UTC)" (scheduled at 2023-01-17 05:28:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:28:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 28, 27, 227281), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:28:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:28:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:28:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:38.735066+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:28:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:38 UTC)" (scheduled at 2023-01-17 05:28:28.735066+00:00) notificationworker stdout | 2023-01-17 05:28:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 28, 735963), True, datetime.datetime(2023, 1, 17, 5, 28, 28, 735963), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:38 UTC)" executed successfully builder stdout | 2023-01-17 05:28:30,253 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:30,262 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:30,262 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:30,268 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:30,302 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:28:30,468 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:28:30,732 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:28:31,573 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:28:31,612 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:28:32,878 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:28:33,618 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:28:33,620 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:28:34,053 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:28:34,938 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:28:35,014 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:28:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:28:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:05.848675+00:00 (in 29.999541 seconds) gcworker stdout | 2023-01-17 05:28:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:05 UTC)" (scheduled at 2023-01-17 05:28:35.848675+00:00) gcworker stdout | 2023-01-17 05:28:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723715849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:28:35,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:28:35,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:28:35,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:28:35,888 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:28:35,891 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:28:35,893 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:28:35,924 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:28:36,664 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:28:36,685 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:28:37,625 [244] [DEBUG] [app] Starting request: urn:request:cf298690-7273-4d8c-9fa9-7572ec16c069 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:37,626 [239] [DEBUG] [app] Starting request: urn:request:e53ceb4a-70a4-4c3f-9814-a3be790344bd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:37,630 [232] [DEBUG] [app] Starting request: urn:request:cb8a103f-be7f-4376-9ed6-856e4a028837 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:37,631 [232] [DEBUG] [app] Ending request: urn:request:cb8a103f-be7f-4376-9ed6-856e4a028837 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb8a103f-be7f-4376-9ed6-856e4a028837', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:37,631 [236] [DEBUG] [app] Starting request: urn:request:456c47a1-70c7-47aa-bfd2-2caab33ab107 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:28:37,631 [236] [DEBUG] [app] Ending request: urn:request:456c47a1-70c7-47aa-bfd2-2caab33ab107 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:456c47a1-70c7-47aa-bfd2-2caab33ab107', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:28:37,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:28:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:37,636 [243] [DEBUG] [app] Starting request: urn:request:e7aa8bc3-d961-4df9-bf96-bb53b5b06f81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:37,636 [241] [DEBUG] [app] Starting request: urn:request:34c9570a-a0f7-48a7-9e81-f22f39ff2649 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:37,636 [243] [DEBUG] [app] Ending request: urn:request:e7aa8bc3-d961-4df9-bf96-bb53b5b06f81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7aa8bc3-d961-4df9-bf96-bb53b5b06f81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:37,636 [241] [DEBUG] [app] Ending request: urn:request:34c9570a-a0f7-48a7-9e81-f22f39ff2649 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:34c9570a-a0f7-48a7-9e81-f22f39ff2649', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:37,647 [244] [DEBUG] [app] Ending request: urn:request:cf298690-7273-4d8c-9fa9-7572ec16c069 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf298690-7273-4d8c-9fa9-7572ec16c069', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:37,647 [239] [DEBUG] [app] Ending request: urn:request:e53ceb4a-70a4-4c3f-9814-a3be790344bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e53ceb4a-70a4-4c3f-9814-a3be790344bd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:28:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) storagereplication stdout | 2023-01-17 05:28:37,950 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:28:38,234 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:28:38,238 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:28:38,242 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:28:38,244 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:28:38,247 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:28:38,250 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:28:38,252 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:28:38,255 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:28:38,257 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:28:38,440 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:28:38,442 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:28:38,445 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:28:38,448 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:28:38,453 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:28:38,549 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:28:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:43.935629+00:00 (in 5.200140 seconds) notificationworker stdout | 2023-01-17 05:28:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:48 UTC)" (scheduled at 2023-01-17 05:28:38.735066+00:00) notificationworker stdout | 2023-01-17 05:28:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 38, 735843), True, datetime.datetime(2023, 1, 17, 5, 28, 38, 735843), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:28:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:28:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:48.738458+00:00 (in 9.707229 seconds) repositorygcworker stdout | 2023-01-17 05:28:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:39 UTC)" (scheduled at 2023-01-17 05:28:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:28:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:28:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 39, 31654), True, datetime.datetime(2023, 1, 17, 5, 28, 39, 31654), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:28:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:28:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:28:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:28:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:52.837365+00:00 (in 13.606907 seconds) securityworker stdout | 2023-01-17 05:28:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:09 UTC)" (scheduled at 2023-01-17 05:28:39.230038+00:00) securityworker stdout | 2023-01-17 05:28:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:28:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:28:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:28:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:28:39,274 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:28:39,320 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:28:39,355 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:28:39,369 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:28:40,278 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:40,287 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:40,287 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:40,293 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:48.735066+00:00 (in 4.798978 seconds) notificationworker stdout | 2023-01-17 05:28:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:43 UTC)" (scheduled at 2023-01-17 05:28:43.935629+00:00) notificationworker stdout | 2023-01-17 05:28:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:28:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:28:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:28:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:59.337413+00:00 (in 11.404461 seconds) namespacegcworker stdout | 2023-01-17 05:28:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:47 UTC)" (scheduled at 2023-01-17 05:28:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:28:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:28:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:56.935055+00:00 (in 8.199486 seconds) notificationworker stdout | 2023-01-17 05:28:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:58 UTC)" (scheduled at 2023-01-17 05:28:48.735066+00:00) notificationworker stdout | 2023-01-17 05:28:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 48, 735982), True, datetime.datetime(2023, 1, 17, 5, 28, 48, 735982), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:28:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:28:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:39.030814+00:00 (in 50.291894 seconds) repositorygcworker stdout | 2023-01-17 05:28:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:48 UTC)" (scheduled at 2023-01-17 05:28:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:28:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:28:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:28:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:28:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:24.641816+00:00 (in 35.811062 seconds) exportactionlogsworker stdout | 2023-01-17 05:28:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:48 UTC)" (scheduled at 2023-01-17 05:28:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:28:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:28:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 48, 831148), True, datetime.datetime(2023, 1, 17, 5, 28, 48, 831148), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:28:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:28:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:28:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:48 UTC)" executed successfully builder stdout | 2023-01-17 05:28:50,303 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:50,312 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:50,312 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:50,317 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:52,625 [244] [DEBUG] [app] Starting request: urn:request:8fac1b99-6023-4b21-9265-1ad4d095f8de (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:52,626 [241] [DEBUG] [app] Starting request: urn:request:a5274978-6fcb-4bc9-9108-aae417ad1851 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:28:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:52,630 [232] [DEBUG] [app] Starting request: urn:request:bc3a672a-3af0-479c-ae5e-285653287b78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:52,631 [236] [DEBUG] [app] Starting request: urn:request:f003c6cf-895d-42ec-9a89-21ae1990ee11 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:52,631 [232] [DEBUG] [app] Ending request: urn:request:bc3a672a-3af0-479c-ae5e-285653287b78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bc3a672a-3af0-479c-ae5e-285653287b78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:28:52,631 [236] [DEBUG] [app] Ending request: urn:request:f003c6cf-895d-42ec-9a89-21ae1990ee11 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f003c6cf-895d-42ec-9a89-21ae1990ee11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:28:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:28:52,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:52,636 [244] [DEBUG] [app] Starting request: urn:request:7ed2706b-6a95-4b53-bb6e-9cbf0529f181 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:52,636 [239] [DEBUG] [app] Starting request: urn:request:4c2f7e79-161c-480a-8605-f54aa20b2a6e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:52,637 [244] [DEBUG] [app] Ending request: urn:request:7ed2706b-6a95-4b53-bb6e-9cbf0529f181 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ed2706b-6a95-4b53-bb6e-9cbf0529f181', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:52,637 [239] [DEBUG] [app] Ending request: urn:request:4c2f7e79-161c-480a-8605-f54aa20b2a6e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c2f7e79-161c-480a-8605-f54aa20b2a6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:28:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:52,648 [241] [DEBUG] [app] Ending request: urn:request:a5274978-6fcb-4bc9-9108-aae417ad1851 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5274978-6fcb-4bc9-9108-aae417ad1851', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:52,648 [244] [DEBUG] [app] Ending request: urn:request:8fac1b99-6023-4b21-9265-1ad4d095f8de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8fac1b99-6023-4b21-9265-1ad4d095f8de', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:52,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:28:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:28:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:28:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:09.230038+00:00 (in 16.392260 seconds) securityworker stdout | 2023-01-17 05:28:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:22 UTC)" (scheduled at 2023-01-17 05:28:52.837365+00:00) securityworker stdout | 2023-01-17 05:28:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:58.735066+00:00 (in 1.799558 seconds) notificationworker stdout | 2023-01-17 05:28:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:33:56 UTC)" (scheduled at 2023-01-17 05:28:56.935055+00:00) notificationworker stdout | 2023-01-17 05:28:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 28, 56, 935835), 'notification/%']) notificationworker stdout | 2023-01-17 05:28:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 28, 56, 935835), True, datetime.datetime(2023, 1, 17, 5, 28, 56, 935835), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:28:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 28, 56, 935835), True, datetime.datetime(2023, 1, 17, 5, 28, 56, 935835), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 28, 56, 935835), 'notification/%']) notificationworker stdout | 2023-01-17 05:28:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:33:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:28:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:28:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:27.226424+00:00 (in 29.999553 seconds) buildlogsarchiver stdout | 2023-01-17 05:28:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:27 UTC)" (scheduled at 2023-01-17 05:28:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:28:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 28, 57, 227217), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:28:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:28:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:28:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:08.735066+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:28:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:08 UTC)" (scheduled at 2023-01-17 05:28:58.735066+00:00) notificationworker stdout | 2023-01-17 05:28:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 58, 735912), True, datetime.datetime(2023, 1, 17, 5, 28, 58, 735912), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:28:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:28:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:15.931994+00:00 (in 16.594122 seconds) namespacegcworker stdout | 2023-01-17 05:28:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:59 UTC)" (scheduled at 2023-01-17 05:28:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:28:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:28:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 59, 338294), True, datetime.datetime(2023, 1, 17, 5, 28, 59, 338294), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:28:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:28:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:28:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:59 UTC)" executed successfully builder stdout | 2023-01-17 05:29:00,328 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:00,337 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:00,337 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics securityworker stdout | 2023-01-17 05:29:00,338 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:29:00,342 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:29:00,502 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:29:00,739 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:29:01,602 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:29:01,647 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:29:02,913 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:29:03,641 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:29:03,652 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:29:04,085 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:29:04,972 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:29:05,045 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:29:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:29:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:35.848675+00:00 (in 29.999516 seconds) gcworker stdout | 2023-01-17 05:29:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:35 UTC)" (scheduled at 2023-01-17 05:29:05.848675+00:00) gcworker stdout | 2023-01-17 05:29:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723745849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:29:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:29:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:29:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:29:05,895 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:29:05,897 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:29:05,899 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:29:05,941 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:29:06,699 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:29:06,713 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:29:07,625 [243] [DEBUG] [app] Starting request: urn:request:6e02391c-57c6-46d2-9544-91d48a07d1de (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:07,625 [239] [DEBUG] [app] Starting request: urn:request:300af49c-d708-419f-aa70-2e62ecfdf2f2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:07,630 [240] [DEBUG] [app] Starting request: urn:request:769c1dcf-9866-4f7a-a2f1-d37f85ea2cd6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:07,630 [242] [DEBUG] [app] Starting request: urn:request:00a43634-a8ff-4d2a-9cba-973bece1b59f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:07,630 [240] [DEBUG] [app] Ending request: urn:request:769c1dcf-9866-4f7a-a2f1-d37f85ea2cd6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:769c1dcf-9866-4f7a-a2f1-d37f85ea2cd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:07,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:29:07,630 [242] [DEBUG] [app] Ending request: urn:request:00a43634-a8ff-4d2a-9cba-973bece1b59f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:00a43634-a8ff-4d2a-9cba-973bece1b59f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:29:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:29:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:07,635 [244] [DEBUG] [app] Starting request: urn:request:2e1c67ee-4cd7-4d56-b8ad-39c2b8007c3d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:07,635 [241] [DEBUG] [app] Starting request: urn:request:311d92fd-9213-4daa-baea-783e60d42e8b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:07,635 [244] [DEBUG] [app] Ending request: urn:request:2e1c67ee-4cd7-4d56-b8ad-39c2b8007c3d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e1c67ee-4cd7-4d56-b8ad-39c2b8007c3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:07,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:07,636 [241] [DEBUG] [app] Ending request: urn:request:311d92fd-9213-4daa-baea-783e60d42e8b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:311d92fd-9213-4daa-baea-783e60d42e8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:07,646 [239] [DEBUG] [app] Ending request: urn:request:300af49c-d708-419f-aa70-2e62ecfdf2f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:300af49c-d708-419f-aa70-2e62ecfdf2f2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:07,646 [243] [DEBUG] [app] Ending request: urn:request:6e02391c-57c6-46d2-9544-91d48a07d1de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e02391c-57c6-46d2-9544-91d48a07d1de', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:07,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:29:07,647 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:29:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:29:07,986 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:29:08,244 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:29:08,246 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:29:08,251 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:29:08,254 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:29:08,256 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:29:08,259 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:29:08,261 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:29:08,264 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:29:08,266 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:29:08,446 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:29:08,449 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:29:08,451 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:29:08,454 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:29:08,459 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:29:08,583 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:29:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:18.735066+00:00 (in 9.999608 seconds) notificationworker stdout | 2023-01-17 05:29:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:18 UTC)" (scheduled at 2023-01-17 05:29:08.735066+00:00) notificationworker stdout | 2023-01-17 05:29:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 8, 735867), True, datetime.datetime(2023, 1, 17, 5, 29, 8, 735867), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:08,749 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:08,749 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:08,749 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:22.837365+00:00 (in 13.606857 seconds) securityworker stdout | 2023-01-17 05:29:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:39 UTC)" (scheduled at 2023-01-17 05:29:09.230038+00:00) securityworker stdout | 2023-01-17 05:29:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:29:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:29:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:29:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:29:09,287 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:29:09,355 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:29:09,369 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:29:09,403 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:29:10,353 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:10,362 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:10,362 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:10,367 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:29:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:29:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:47.932490+00:00 (in 32.000005 seconds) namespacegcworker stdout | 2023-01-17 05:29:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:34:15 UTC)" (scheduled at 2023-01-17 05:29:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:29:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 29, 15, 932843), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:29:15,939 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 29, 15, 932843), True, datetime.datetime(2023, 1, 17, 5, 29, 15, 932843), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:29:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 29, 15, 932843), True, datetime.datetime(2023, 1, 17, 5, 29, 15, 932843), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 29, 15, 932843), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:29:15,943 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:29:15,943 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:34:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:28.735066+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:29:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:28 UTC)" (scheduled at 2023-01-17 05:29:18.735066+00:00) notificationworker stdout | 2023-01-17 05:29:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 18, 735963), True, datetime.datetime(2023, 1, 17, 5, 29, 18, 735963), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:29:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:29:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:25.334138+00:00 (in 5.091458 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:29:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:20 UTC)" (scheduled at 2023-01-17 05:29:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:29:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:29:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 20, 243099), True, datetime.datetime(2023, 1, 17, 5, 29, 20, 243099), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:29:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:29:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:29:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:20 UTC)" executed successfully builder stdout | 2023-01-17 05:29:20,377 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:20,386 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:20,386 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:20,392 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:22,625 [243] [DEBUG] [app] Starting request: urn:request:9b414a03-4a31-40cc-ab91-945f37e5d490 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:22,625 [241] [DEBUG] [app] Starting request: urn:request:176784e0-b7cb-4252-b83d-e916fde8bd37 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:22,630 [232] [DEBUG] [app] Starting request: urn:request:636d72b3-a69d-4340-b18a-2cd8bdf09c65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:22,630 [242] [DEBUG] [app] Starting request: urn:request:29f36497-45a2-4e05-86f2-461508d5f9bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:22,630 [232] [DEBUG] [app] Ending request: urn:request:636d72b3-a69d-4340-b18a-2cd8bdf09c65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:636d72b3-a69d-4340-b18a-2cd8bdf09c65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:22,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:29:22,631 [242] [DEBUG] [app] Ending request: urn:request:29f36497-45a2-4e05-86f2-461508d5f9bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29f36497-45a2-4e05-86f2-461508d5f9bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:29:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:29:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:29:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:22,635 [239] [DEBUG] [app] Starting request: urn:request:8b50f815-a50d-433e-a4d7-edd0e8f87c46 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:22,635 [241] [DEBUG] [app] Starting request: urn:request:42f667ca-3b4a-454f-8c94-09250d48cd22 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:22,636 [239] [DEBUG] [app] Ending request: urn:request:8b50f815-a50d-433e-a4d7-edd0e8f87c46 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b50f815-a50d-433e-a4d7-edd0e8f87c46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:22,636 [241] [DEBUG] [app] Ending request: urn:request:42f667ca-3b4a-454f-8c94-09250d48cd22 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:42f667ca-3b4a-454f-8c94-09250d48cd22', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:22,654 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:22,654 [243] [DEBUG] [app] Ending request: urn:request:9b414a03-4a31-40cc-ab91-945f37e5d490 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b414a03-4a31-40cc-ab91-945f37e5d490', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:22,654 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:22,654 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:29:22,659 [241] [DEBUG] [app] Ending request: urn:request:176784e0-b7cb-4252-b83d-e916fde8bd37 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:176784e0-b7cb-4252-b83d-e916fde8bd37', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:22,659 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:22,659 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) securityworker stdout | 2023-01-17 05:29:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:39.230038+00:00 (in 16.392169 seconds) securityworker stdout | 2023-01-17 05:29:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:52 UTC)" (scheduled at 2023-01-17 05:29:22.837365+00:00) securityworker stdout | 2023-01-17 05:29:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:29:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:29:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:48.830274+00:00 (in 24.188005 seconds) exportactionlogsworker stdout | 2023-01-17 05:29:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:24 UTC)" (scheduled at 2023-01-17 05:29:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:29:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:29:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:29:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:29:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:20.242198+00:00 (in 54.907615 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:29:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:25 UTC)" (scheduled at 2023-01-17 05:29:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:29:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:29:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:29:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:29:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:57.226424+00:00 (in 29.999548 seconds) buildlogsarchiver stdout | 2023-01-17 05:29:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:57 UTC)" (scheduled at 2023-01-17 05:29:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:29:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 29, 27, 227261), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:29:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:29:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:29:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:38.735066+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:29:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:38 UTC)" (scheduled at 2023-01-17 05:29:28.735066+00:00) notificationworker stdout | 2023-01-17 05:29:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 28, 736000), True, datetime.datetime(2023, 1, 17, 5, 29, 28, 736000), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:30,364 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:29:30,402 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:30,411 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:30,411 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:30,416 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:29:30,509 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:29:30,775 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:29:31,637 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:29:31,678 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:29:32,919 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:29:33,672 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:29:33,676 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:29:34,121 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:29:35,007 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:29:35,079 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:29:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:29:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:05.848675+00:00 (in 29.999431 seconds) gcworker stdout | 2023-01-17 05:29:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:05 UTC)" (scheduled at 2023-01-17 05:29:35.848675+00:00) gcworker stdout | 2023-01-17 05:29:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723775849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:29:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:29:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:29:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:29:35,902 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:29:35,905 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:29:35,907 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:29:35,976 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:29:36,735 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:29:36,740 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:29:37,625 [243] [DEBUG] [app] Starting request: urn:request:cde11135-7b7c-4574-9ea7-247ae2e1864d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:37,626 [244] [DEBUG] [app] Starting request: urn:request:3bad16b5-b0fc-4f5e-b43a-8df9fdd81872 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:37,631 [240] [DEBUG] [app] Starting request: urn:request:57e7b8b2-d654-4a25-97a6-6cec77930af3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,631 [235] [DEBUG] [app] Starting request: urn:request:ac18292a-adf1-4b79-85f8-166e2a8a0a12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,631 [240] [DEBUG] [app] Ending request: urn:request:57e7b8b2-d654-4a25-97a6-6cec77930af3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57e7b8b2-d654-4a25-97a6-6cec77930af3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:29:37,632 [235] [DEBUG] [app] Ending request: urn:request:ac18292a-adf1-4b79-85f8-166e2a8a0a12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac18292a-adf1-4b79-85f8-166e2a8a0a12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:29:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:29:37,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:29:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:37,636 [241] [DEBUG] [app] Starting request: urn:request:286bc382-548f-4d50-b66a-3090a9367fb4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:37,636 [239] [DEBUG] [app] Starting request: urn:request:be4f79f9-5bde-4dc6-9502-27ab6c8d1725 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:37,637 [241] [DEBUG] [app] Ending request: urn:request:286bc382-548f-4d50-b66a-3090a9367fb4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:286bc382-548f-4d50-b66a-3090a9367fb4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:37,637 [239] [DEBUG] [app] Ending request: urn:request:be4f79f9-5bde-4dc6-9502-27ab6c8d1725 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be4f79f9-5bde-4dc6-9502-27ab6c8d1725', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:37,647 [243] [DEBUG] [app] Ending request: urn:request:cde11135-7b7c-4574-9ea7-247ae2e1864d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cde11135-7b7c-4574-9ea7-247ae2e1864d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:37,648 [244] [DEBUG] [app] Ending request: urn:request:3bad16b5-b0fc-4f5e-b43a-8df9fdd81872 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3bad16b5-b0fc-4f5e-b43a-8df9fdd81872', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:29:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) storagereplication stdout | 2023-01-17 05:29:38,013 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:29:38,251 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:29:38,254 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:29:38,260 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:29:38,262 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:29:38,264 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:29:38,267 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:29:38,269 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:29:38,273 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:29:38,275 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:29:38,452 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:29:38,455 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:29:38,457 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:29:38,460 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:29:38,465 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:29:38,590 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:29:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:43.935629+00:00 (in 5.200156 seconds) notificationworker stdout | 2023-01-17 05:29:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:48 UTC)" (scheduled at 2023-01-17 05:29:38.735066+00:00) notificationworker stdout | 2023-01-17 05:29:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 38, 735814), True, datetime.datetime(2023, 1, 17, 5, 29, 38, 735814), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:29:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:29:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:48.738458+00:00 (in 9.707221 seconds) repositorygcworker stdout | 2023-01-17 05:29:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:39 UTC)" (scheduled at 2023-01-17 05:29:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:29:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:29:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 39, 31651), True, datetime.datetime(2023, 1, 17, 5, 29, 39, 31651), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:29:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:29:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:29:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:52.837365+00:00 (in 13.606880 seconds) securityworker stdout | 2023-01-17 05:29:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:09 UTC)" (scheduled at 2023-01-17 05:29:39.230038+00:00) securityworker stdout | 2023-01-17 05:29:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:29:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:29:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:39,247 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:39,247 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:29:39,247 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:29:39,248 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:29:39,248 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:39,248 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:39,248 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:39,248 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:29:39,248 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:29:39,324 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:29:39,381 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:29:39,403 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:29:39,428 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:29:40,427 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:40,436 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:40,436 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:40,441 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:48.735066+00:00 (in 4.799020 seconds) notificationworker stdout | 2023-01-17 05:29:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:43 UTC)" (scheduled at 2023-01-17 05:29:43.935629+00:00) notificationworker stdout | 2023-01-17 05:29:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:29:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:29:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:29:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:59.337413+00:00 (in 11.404439 seconds) namespacegcworker stdout | 2023-01-17 05:29:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:47 UTC)" (scheduled at 2023-01-17 05:29:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:29:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:29:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:58.735066+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 05:29:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:58 UTC)" (scheduled at 2023-01-17 05:29:48.735066+00:00) notificationworker stdout | 2023-01-17 05:29:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 48, 735951), True, datetime.datetime(2023, 1, 17, 5, 29, 48, 735951), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:29:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:29:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:39.030814+00:00 (in 50.291921 seconds) repositorygcworker stdout | 2023-01-17 05:29:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:48 UTC)" (scheduled at 2023-01-17 05:29:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:29:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:29:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:29:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:29:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:24.641816+00:00 (in 35.811106 seconds) exportactionlogsworker stdout | 2023-01-17 05:29:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:48 UTC)" (scheduled at 2023-01-17 05:29:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:29:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:29:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 48, 831130), True, datetime.datetime(2023, 1, 17, 5, 29, 48, 831130), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:29:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:29:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:29:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:48 UTC)" executed successfully builder stdout | 2023-01-17 05:29:50,452 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:50,460 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:50,460 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:50,466 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:52,626 [243] [DEBUG] [app] Starting request: urn:request:3500d5b3-6b53-411d-8467-e25c7ef77966 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:52,626 [239] [DEBUG] [app] Starting request: urn:request:6d7dccb3-2497-45d7-a8f4-33095e659df7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:29:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:52,630 [236] [DEBUG] [app] Starting request: urn:request:f3bac9cf-cede-418b-a2d6-686ecd03f3f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:52,631 [235] [DEBUG] [app] Starting request: urn:request:05c8ce6b-cab0-4394-b167-10907544bfc9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:52,631 [236] [DEBUG] [app] Ending request: urn:request:f3bac9cf-cede-418b-a2d6-686ecd03f3f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3bac9cf-cede-418b-a2d6-686ecd03f3f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:52,631 [235] [DEBUG] [app] Ending request: urn:request:05c8ce6b-cab0-4394-b167-10907544bfc9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:05c8ce6b-cab0-4394-b167-10907544bfc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:29:52,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:29:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:52,635 [244] [DEBUG] [app] Starting request: urn:request:3b30faf1-22fe-417f-b5cf-ce3a63db92c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:52,636 [244] [DEBUG] [app] Ending request: urn:request:3b30faf1-22fe-417f-b5cf-ce3a63db92c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b30faf1-22fe-417f-b5cf-ce3a63db92c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:29:52,637 [241] [DEBUG] [app] Starting request: urn:request:541216c3-be1d-4ae6-b020-b4e20cfece1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:52,637 [241] [DEBUG] [app] Ending request: urn:request:541216c3-be1d-4ae6-b020-b4e20cfece1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:541216c3-be1d-4ae6-b020-b4e20cfece1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:52,647 [239] [DEBUG] [app] Ending request: urn:request:6d7dccb3-2497-45d7-a8f4-33095e659df7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d7dccb3-2497-45d7-a8f4-33095e659df7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:29:52,648 [243] [DEBUG] [app] Ending request: urn:request:3500d5b3-6b53-411d-8467-e25c7ef77966 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3500d5b3-6b53-411d-8467-e25c7ef77966', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:52,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:29:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:29:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:09.230038+00:00 (in 16.392227 seconds) securityworker stdout | 2023-01-17 05:29:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:22 UTC)" (scheduled at 2023-01-17 05:29:52.837365+00:00) securityworker stdout | 2023-01-17 05:29:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:29:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:29:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:27.226424+00:00 (in 29.999553 seconds) buildlogsarchiver stdout | 2023-01-17 05:29:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:27 UTC)" (scheduled at 2023-01-17 05:29:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:29:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 29, 57, 227211), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:29:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:29:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:29:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:08.735066+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:29:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:08 UTC)" (scheduled at 2023-01-17 05:29:58.735066+00:00) notificationworker stdout | 2023-01-17 05:29:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 58, 735896), True, datetime.datetime(2023, 1, 17, 5, 29, 58, 735896), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:29:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:29:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:47.932490+00:00 (in 48.594597 seconds) namespacegcworker stdout | 2023-01-17 05:29:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:59 UTC)" (scheduled at 2023-01-17 05:29:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:29:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:29:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 59, 338324), True, datetime.datetime(2023, 1, 17, 5, 29, 59, 338324), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:29:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:29:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:29:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:00,385 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:30:00,469 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:00,478 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:00,478 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:00,483 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:30:00,539 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:30:00,811 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:30:01,672 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:30:01,697 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:30:02,955 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:30:03,707 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:30:03,710 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:30:04,130 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:30:05,041 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:30:05,114 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:30:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:30:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:35.848675+00:00 (in 29.999503 seconds) gcworker stdout | 2023-01-17 05:30:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:35 UTC)" (scheduled at 2023-01-17 05:30:05.848675+00:00) gcworker stdout | 2023-01-17 05:30:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723805849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:30:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:30:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:30:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:30:05,909 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:30:05,911 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:30:05,913 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:30:06,011 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:30:06,763 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:30:06,776 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:30:07,626 [243] [DEBUG] [app] Starting request: urn:request:fac7b0bd-2a0c-4d74-a6fb-85664ca3effe (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:07,626 [239] [DEBUG] [app] Starting request: urn:request:19f41a23-1a58-48c7-8621-2ef62e952d2e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:07,630 [232] [DEBUG] [app] Starting request: urn:request:b8569917-1146-4789-94e8-914ab6b4f0a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:07,631 [242] [DEBUG] [app] Starting request: urn:request:2aba3465-e727-463c-b63f-e39c3214ea73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:07,631 [232] [DEBUG] [app] Ending request: urn:request:b8569917-1146-4789-94e8-914ab6b4f0a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b8569917-1146-4789-94e8-914ab6b4f0a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:07,631 [242] [DEBUG] [app] Ending request: urn:request:2aba3465-e727-463c-b63f-e39c3214ea73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2aba3465-e727-463c-b63f-e39c3214ea73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:30:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:30:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:07,635 [244] [DEBUG] [app] Starting request: urn:request:df02d303-22e0-4f61-a65f-411818b38f32 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:07,636 [244] [DEBUG] [app] Ending request: urn:request:df02d303-22e0-4f61-a65f-411818b38f32 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df02d303-22e0-4f61-a65f-411818b38f32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:30:07,636 [241] [DEBUG] [app] Starting request: urn:request:7b0fc180-2dde-4694-a38a-b54dcb9e3690 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:07,637 [241] [DEBUG] [app] Ending request: urn:request:7b0fc180-2dde-4694-a38a-b54dcb9e3690 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7b0fc180-2dde-4694-a38a-b54dcb9e3690', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:30:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:07,647 [239] [DEBUG] [app] Ending request: urn:request:19f41a23-1a58-48c7-8621-2ef62e952d2e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:19f41a23-1a58-48c7-8621-2ef62e952d2e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:30:07,648 [243] [DEBUG] [app] Ending request: urn:request:fac7b0bd-2a0c-4d74-a6fb-85664ca3effe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fac7b0bd-2a0c-4d74-a6fb-85664ca3effe', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:07,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) storagereplication stdout | 2023-01-17 05:30:08,031 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:30:08,260 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:30:08,262 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:30:08,268 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:30:08,271 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:30:08,273 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:30:08,276 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:30:08,279 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:30:08,281 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:30:08,284 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:30:08,458 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:30:08,462 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:30:08,464 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:30:08,467 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:30:08,471 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:30:08,625 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:30:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:18.735066+00:00 (in 9.999612 seconds) notificationworker stdout | 2023-01-17 05:30:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:18 UTC)" (scheduled at 2023-01-17 05:30:08.735066+00:00) notificationworker stdout | 2023-01-17 05:30:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 8, 735865), True, datetime.datetime(2023, 1, 17, 5, 30, 8, 735865), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:22.837365+00:00 (in 13.606895 seconds) securityworker stdout | 2023-01-17 05:30:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:39 UTC)" (scheduled at 2023-01-17 05:30:09.230038+00:00) securityworker stdout | 2023-01-17 05:30:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:30:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:30:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:30:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:30:09,347 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:30:09,416 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:30:09,418 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:30:09,453 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:30:10,494 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:10,502 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:10,502 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:10,508 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:28.735066+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 05:30:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:28 UTC)" (scheduled at 2023-01-17 05:30:18.735066+00:00) notificationworker stdout | 2023-01-17 05:30:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 18, 735981), True, datetime.datetime(2023, 1, 17, 5, 30, 18, 735981), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:30:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:30:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:25.334138+00:00 (in 5.091508 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:30:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:20 UTC)" (scheduled at 2023-01-17 05:30:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:30:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:30:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 20, 243024), True, datetime.datetime(2023, 1, 17, 5, 30, 20, 243024), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:30:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:30:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:30:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:20 UTC)" executed successfully builder stdout | 2023-01-17 05:30:20,518 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:20,527 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:20,527 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:20,533 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:22,625 [239] [DEBUG] [app] Starting request: urn:request:3df6a2e6-a378-440d-83a5-5b7eb8c7e12e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:22,626 [244] [DEBUG] [app] Starting request: urn:request:97654928-2c31-49bf-9b7e-0b0fabc81041 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:22,630 [232] [DEBUG] [app] Starting request: urn:request:9857cd95-acab-4ccf-8e88-886aaeee3c50 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:22,631 [237] [DEBUG] [app] Starting request: urn:request:e3473d1f-f7fb-4352-b8fb-6101188f6ef2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:22,631 [232] [DEBUG] [app] Ending request: urn:request:9857cd95-acab-4ccf-8e88-886aaeee3c50 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9857cd95-acab-4ccf-8e88-886aaeee3c50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:30:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:30:22,631 [237] [DEBUG] [app] Ending request: urn:request:e3473d1f-f7fb-4352-b8fb-6101188f6ef2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e3473d1f-f7fb-4352-b8fb-6101188f6ef2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:30:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:30:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:22,635 [239] [DEBUG] [app] Starting request: urn:request:7e1f80cf-65e3-4839-91d0-ee09608bb5ab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:22,635 [239] [DEBUG] [app] Ending request: urn:request:7e1f80cf-65e3-4839-91d0-ee09608bb5ab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e1f80cf-65e3-4839-91d0-ee09608bb5ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:22,637 [243] [DEBUG] [app] Starting request: urn:request:6e54169b-dab5-4d06-8e1d-313a2cc4d0f3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:22,637 [243] [DEBUG] [app] Ending request: urn:request:6e54169b-dab5-4d06-8e1d-313a2cc4d0f3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e54169b-dab5-4d06-8e1d-313a2cc4d0f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:30:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:22,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:22,647 [244] [DEBUG] [app] Ending request: urn:request:97654928-2c31-49bf-9b7e-0b0fabc81041 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97654928-2c31-49bf-9b7e-0b0fabc81041', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:30:22,648 [239] [DEBUG] [app] Ending request: urn:request:3df6a2e6-a378-440d-83a5-5b7eb8c7e12e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3df6a2e6-a378-440d-83a5-5b7eb8c7e12e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:30:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:39.230038+00:00 (in 16.392246 seconds) securityworker stdout | 2023-01-17 05:30:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:52 UTC)" (scheduled at 2023-01-17 05:30:22.837365+00:00) securityworker stdout | 2023-01-17 05:30:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:30:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:30:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:48.830274+00:00 (in 24.187945 seconds) exportactionlogsworker stdout | 2023-01-17 05:30:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:24 UTC)" (scheduled at 2023-01-17 05:30:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:30:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:30:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:30:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:30:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:20.242198+00:00 (in 54.907598 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:30:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:25 UTC)" (scheduled at 2023-01-17 05:30:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:30:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:30:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:30:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:30:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:57.226424+00:00 (in 29.999551 seconds) buildlogsarchiver stdout | 2023-01-17 05:30:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:57 UTC)" (scheduled at 2023-01-17 05:30:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:30:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 30, 27, 227246), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:30:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:30:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:30:27,236 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:38.735066+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 05:30:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:38 UTC)" (scheduled at 2023-01-17 05:30:28.735066+00:00) notificationworker stdout | 2023-01-17 05:30:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 28, 735964), True, datetime.datetime(2023, 1, 17, 5, 30, 28, 735964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:30,409 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:30:30,543 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue exportactionlogsworker stdout | 2023-01-17 05:30:30,549 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:30:30,552 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:30,552 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:30,557 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:30:30,847 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:30:31,681 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:30:31,703 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:30:32,978 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:30:33,744 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:30:33,746 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:30:34,166 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:30:35,077 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:30:35,136 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:30:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:30:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:05.848675+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:30:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:05 UTC)" (scheduled at 2023-01-17 05:30:35.848675+00:00) gcworker stdout | 2023-01-17 05:30:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723835849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:30:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:30:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:30:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:30:35,916 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:30:35,918 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:30:35,920 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:30:36,045 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:30:36,800 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:30:36,803 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:30:37,625 [244] [DEBUG] [app] Starting request: urn:request:7dac5406-c643-4635-91b9-b4cfc768bc25 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:37,625 [241] [DEBUG] [app] Starting request: urn:request:5a7ae5fe-4cf4-4f46-80e3-b4e86c65d214 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:37,630 [240] [DEBUG] [app] Starting request: urn:request:99969181-0531-4c31-bbcd-cd74c2ea2ea7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:37,630 [232] [DEBUG] [app] Starting request: urn:request:cc0be290-cab1-4a73-9dd0-b6be0fa8e46f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:37,631 [240] [DEBUG] [app] Ending request: urn:request:99969181-0531-4c31-bbcd-cd74c2ea2ea7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99969181-0531-4c31-bbcd-cd74c2ea2ea7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:30:37,631 [232] [DEBUG] [app] Ending request: urn:request:cc0be290-cab1-4a73-9dd0-b6be0fa8e46f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc0be290-cab1-4a73-9dd0-b6be0fa8e46f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:30:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:30:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:30:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:37,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:37,635 [243] [DEBUG] [app] Starting request: urn:request:be0cf755-b212-4add-bb52-535114aa3999 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:37,636 [243] [DEBUG] [app] Ending request: urn:request:be0cf755-b212-4add-bb52-535114aa3999 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be0cf755-b212-4add-bb52-535114aa3999', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:37,636 [239] [DEBUG] [app] Starting request: urn:request:228d64d8-04ef-4db8-9441-5c03dcfa5a1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:37,636 [239] [DEBUG] [app] Ending request: urn:request:228d64d8-04ef-4db8-9441-5c03dcfa5a1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:228d64d8-04ef-4db8-9441-5c03dcfa5a1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:37,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:37,648 [244] [DEBUG] [app] Ending request: urn:request:7dac5406-c643-4635-91b9-b4cfc768bc25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7dac5406-c643-4635-91b9-b4cfc768bc25', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:37,648 [241] [DEBUG] [app] Ending request: urn:request:5a7ae5fe-4cf4-4f46-80e3-b4e86c65d214 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5a7ae5fe-4cf4-4f46-80e3-b4e86c65d214', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:30:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) storagereplication stdout | 2023-01-17 05:30:38,039 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:30:38,269 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:30:38,272 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:30:38,275 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:30:38,278 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:30:38,280 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:30:38,283 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:30:38,285 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:30:38,287 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:30:38,291 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:30:38,464 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:30:38,468 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:30:38,472 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:30:38,474 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:30:38,477 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:30:38,661 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:30:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:43.935629+00:00 (in 5.200164 seconds) notificationworker stdout | 2023-01-17 05:30:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:48 UTC)" (scheduled at 2023-01-17 05:30:38.735066+00:00) notificationworker stdout | 2023-01-17 05:30:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 38, 735809), True, datetime.datetime(2023, 1, 17, 5, 30, 38, 735809), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:30:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:30:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:48.738458+00:00 (in 9.707217 seconds) repositorygcworker stdout | 2023-01-17 05:30:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:39 UTC)" (scheduled at 2023-01-17 05:30:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:30:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:30:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 39, 31639), True, datetime.datetime(2023, 1, 17, 5, 30, 39, 31639), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:30:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:30:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:30:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:52.837365+00:00 (in 13.606902 seconds) securityworker stdout | 2023-01-17 05:30:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:09 UTC)" (scheduled at 2023-01-17 05:30:39.230038+00:00) securityworker stdout | 2023-01-17 05:30:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:30:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:30:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:30:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:30:39,381 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:30:39,451 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:30:39,453 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:30:39,482 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:30:40,568 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:40,576 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:40,576 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:40,582 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:48.735066+00:00 (in 4.799003 seconds) notificationworker stdout | 2023-01-17 05:30:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:43 UTC)" (scheduled at 2023-01-17 05:30:43.935629+00:00) notificationworker stdout | 2023-01-17 05:30:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:30:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:30:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:30:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:59.337413+00:00 (in 11.404441 seconds) namespacegcworker stdout | 2023-01-17 05:30:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:47 UTC)" (scheduled at 2023-01-17 05:30:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:30:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:30:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:58.735066+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:30:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:58 UTC)" (scheduled at 2023-01-17 05:30:48.735066+00:00) notificationworker stdout | 2023-01-17 05:30:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 48, 735892), True, datetime.datetime(2023, 1, 17, 5, 30, 48, 735892), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:30:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:30:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:39.030814+00:00 (in 50.291928 seconds) repositorygcworker stdout | 2023-01-17 05:30:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:48 UTC)" (scheduled at 2023-01-17 05:30:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:30:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:30:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:30:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:30:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:24.641816+00:00 (in 35.811077 seconds) exportactionlogsworker stdout | 2023-01-17 05:30:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:48 UTC)" (scheduled at 2023-01-17 05:30:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:30:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:30:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 48, 831139), True, datetime.datetime(2023, 1, 17, 5, 30, 48, 831139), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:30:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:30:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:30:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:48 UTC)" executed successfully builder stdout | 2023-01-17 05:30:50,584 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:50,593 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:50,593 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:50,599 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:52,625 [243] [DEBUG] [app] Starting request: urn:request:298a9c9f-7fa0-456c-a791-a3d06b818bba (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:52,625 [241] [DEBUG] [app] Starting request: urn:request:4aa8bcca-b880-4d5f-ae24-62fc3c2bebbc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:30:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:52,630 [242] [DEBUG] [app] Starting request: urn:request:85b804d4-dd4e-4d02-9d25-aba2b6e6d6ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:52,630 [237] [DEBUG] [app] Starting request: urn:request:e79cf0dc-f81e-486e-8632-77a5712cfd75 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:52,630 [242] [DEBUG] [app] Ending request: urn:request:85b804d4-dd4e-4d02-9d25-aba2b6e6d6ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85b804d4-dd4e-4d02-9d25-aba2b6e6d6ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:30:52,631 [237] [DEBUG] [app] Ending request: urn:request:e79cf0dc-f81e-486e-8632-77a5712cfd75 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e79cf0dc-f81e-486e-8632-77a5712cfd75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:30:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:30:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:52,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:52,635 [239] [DEBUG] [app] Starting request: urn:request:8af35734-e593-4494-9c85-ed1fd3747c24 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:52,635 [243] [DEBUG] [app] Starting request: urn:request:71ba0c82-6c36-48e6-b0d0-a9bd0b1c1742 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:52,635 [239] [DEBUG] [app] Ending request: urn:request:8af35734-e593-4494-9c85-ed1fd3747c24 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8af35734-e593-4494-9c85-ed1fd3747c24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:52,635 [243] [DEBUG] [app] Ending request: urn:request:71ba0c82-6c36-48e6-b0d0-a9bd0b1c1742 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71ba0c82-6c36-48e6-b0d0-a9bd0b1c1742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:52,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:52,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:52,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:52,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:52,646 [243] [DEBUG] [app] Ending request: urn:request:298a9c9f-7fa0-456c-a791-a3d06b818bba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:298a9c9f-7fa0-456c-a791-a3d06b818bba', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:52,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:52,646 [241] [DEBUG] [app] Ending request: urn:request:4aa8bcca-b880-4d5f-ae24-62fc3c2bebbc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4aa8bcca-b880-4d5f-ae24-62fc3c2bebbc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:52,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:30:52,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:30:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:09.230038+00:00 (in 16.392233 seconds) securityworker stdout | 2023-01-17 05:30:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:22 UTC)" (scheduled at 2023-01-17 05:30:52.837365+00:00) securityworker stdout | 2023-01-17 05:30:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:30:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:30:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:27.226424+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 05:30:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:27 UTC)" (scheduled at 2023-01-17 05:30:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:30:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 30, 57, 227297), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:30:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:30:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:30:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:08.735066+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:30:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:08 UTC)" (scheduled at 2023-01-17 05:30:58.735066+00:00) notificationworker stdout | 2023-01-17 05:30:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 58, 735906), True, datetime.datetime(2023, 1, 17, 5, 30, 58, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:30:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:30:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:47.932490+00:00 (in 48.594614 seconds) namespacegcworker stdout | 2023-01-17 05:30:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:59 UTC)" (scheduled at 2023-01-17 05:30:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:30:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:30:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 59, 338285), True, datetime.datetime(2023, 1, 17, 5, 30, 59, 338285), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:30:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:30:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:30:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:00,419 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:31:00,572 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:31:00,609 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:00,618 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:00,618 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:00,624 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:31:00,882 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:31:01,700 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:31:01,730 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:31:03,011 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:31:03,756 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:31:03,759 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:31:04,194 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:31:05,113 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:31:05,144 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:31:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:31:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:35.848675+00:00 (in 29.999544 seconds) gcworker stdout | 2023-01-17 05:31:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:35 UTC)" (scheduled at 2023-01-17 05:31:05.848675+00:00) gcworker stdout | 2023-01-17 05:31:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723865849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:31:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:31:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:31:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:31:05,923 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:31:05,926 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:31:05,928 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:31:06,078 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:31:06,834 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:31:06,836 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:31:07,625 [244] [DEBUG] [app] Starting request: urn:request:3ef3bc57-86c1-4954-90bf-3f9ff690ce98 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:07,625 [239] [DEBUG] [app] Starting request: urn:request:6d784016-8f2d-4109-8f32-fe56f058b4e6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:07,629 [232] [DEBUG] [app] Starting request: urn:request:09344558-f9d5-44bb-a404-ba56bac4d9c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,630 [232] [DEBUG] [app] Ending request: urn:request:09344558-f9d5-44bb-a404-ba56bac4d9c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09344558-f9d5-44bb-a404-ba56bac4d9c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,630 [242] [DEBUG] [app] Starting request: urn:request:1588dd72-0136-42c5-b717-dd1a888fbdae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:31:07,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:31:07,630 [242] [DEBUG] [app] Ending request: urn:request:1588dd72-0136-42c5-b717-dd1a888fbdae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1588dd72-0136-42c5-b717-dd1a888fbdae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:31:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:07,635 [243] [DEBUG] [app] Starting request: urn:request:41a7a872-fead-49f6-a9cb-7a0772f889dc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:07,635 [243] [DEBUG] [app] Ending request: urn:request:41a7a872-fead-49f6-a9cb-7a0772f889dc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:41a7a872-fead-49f6-a9cb-7a0772f889dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:07,635 [241] [DEBUG] [app] Starting request: urn:request:93bb969f-4f6e-4199-98d7-c461c1346e3d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:07,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:07,636 [241] [DEBUG] [app] Ending request: urn:request:93bb969f-4f6e-4199-98d7-c461c1346e3d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93bb969f-4f6e-4199-98d7-c461c1346e3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:31:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:07,646 [239] [DEBUG] [app] Ending request: urn:request:6d784016-8f2d-4109-8f32-fe56f058b4e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d784016-8f2d-4109-8f32-fe56f058b4e6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:07,647 [244] [DEBUG] [app] Ending request: urn:request:3ef3bc57-86c1-4954-90bf-3f9ff690ce98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ef3bc57-86c1-4954-90bf-3f9ff690ce98', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:07,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:31:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:31:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:31:08,053 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:31:08,278 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:31:08,281 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:31:08,284 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:31:08,286 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:31:08,288 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:31:08,291 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:31:08,293 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:31:08,295 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:31:08,299 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:31:08,471 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:31:08,473 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:31:08,478 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:31:08,480 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:31:08,482 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:31:08,696 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:31:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:18.735066+00:00 (in 9.999636 seconds) notificationworker stdout | 2023-01-17 05:31:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:18 UTC)" (scheduled at 2023-01-17 05:31:08.735066+00:00) notificationworker stdout | 2023-01-17 05:31:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 8, 735794), True, datetime.datetime(2023, 1, 17, 5, 31, 8, 735794), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:22.837365+00:00 (in 13.606891 seconds) securityworker stdout | 2023-01-17 05:31:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:39 UTC)" (scheduled at 2023-01-17 05:31:09.230038+00:00) securityworker stdout | 2023-01-17 05:31:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:31:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:31:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:31:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:31:09,417 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:31:09,460 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:31:09,468 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:31:09,516 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:31:10,627 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:10,636 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:10,636 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:10,642 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:28.735066+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 05:31:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:28 UTC)" (scheduled at 2023-01-17 05:31:18.735066+00:00) notificationworker stdout | 2023-01-17 05:31:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 18, 735976), True, datetime.datetime(2023, 1, 17, 5, 31, 18, 735976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:31:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:31:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:25.334138+00:00 (in 5.091472 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:31:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:20 UTC)" (scheduled at 2023-01-17 05:31:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:31:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:31:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 20, 243064), True, datetime.datetime(2023, 1, 17, 5, 31, 20, 243064), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:31:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:31:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:31:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:20 UTC)" executed successfully builder stdout | 2023-01-17 05:31:20,652 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:20,661 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:20,661 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:20,667 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:22,625 [244] [DEBUG] [app] Starting request: urn:request:e5b9a42f-37eb-4851-8c7e-94ddcf222bf4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:22,625 [243] [DEBUG] [app] Starting request: urn:request:4e06ab08-8469-4fa5-a35f-731136f98ccb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:22,630 [242] [DEBUG] [app] Starting request: urn:request:bd60fa52-a64b-49b0-9ab9-14a737988b11 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:22,630 [234] [DEBUG] [app] Starting request: urn:request:08c9e592-bdda-41dd-a1bd-aa4ad911a339 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:22,631 [242] [DEBUG] [app] Ending request: urn:request:bd60fa52-a64b-49b0-9ab9-14a737988b11 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd60fa52-a64b-49b0-9ab9-14a737988b11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:31:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:31:22,631 [234] [DEBUG] [app] Ending request: urn:request:08c9e592-bdda-41dd-a1bd-aa4ad911a339 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08c9e592-bdda-41dd-a1bd-aa4ad911a339', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:31:22,631 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:22,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:22,636 [241] [DEBUG] [app] Starting request: urn:request:7af76dcb-6108-4d22-9cc3-bc2e4829a85a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:22,636 [241] [DEBUG] [app] Ending request: urn:request:7af76dcb-6108-4d22-9cc3-bc2e4829a85a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7af76dcb-6108-4d22-9cc3-bc2e4829a85a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:22,636 [239] [DEBUG] [app] Starting request: urn:request:ef10b4a1-f17b-42c0-b851-090ab1c080d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:22,637 [239] [DEBUG] [app] Ending request: urn:request:ef10b4a1-f17b-42c0-b851-090ab1c080d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef10b4a1-f17b-42c0-b851-090ab1c080d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:31:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:22,648 [244] [DEBUG] [app] Ending request: urn:request:e5b9a42f-37eb-4851-8c7e-94ddcf222bf4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5b9a42f-37eb-4851-8c7e-94ddcf222bf4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:22,648 [243] [DEBUG] [app] Ending request: urn:request:4e06ab08-8469-4fa5-a35f-731136f98ccb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4e06ab08-8469-4fa5-a35f-731136f98ccb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:31:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:31:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:39.230038+00:00 (in 16.392225 seconds) securityworker stdout | 2023-01-17 05:31:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:52 UTC)" (scheduled at 2023-01-17 05:31:22.837365+00:00) securityworker stdout | 2023-01-17 05:31:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:31:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:31:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:44.641497+00:00 (in 19.999209 seconds) exportactionlogsworker stdout | 2023-01-17 05:31:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:24 UTC)" (scheduled at 2023-01-17 05:31:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:31:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:31:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:31:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:31:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:39.333860+00:00 (in 13.999246 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:31:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:25 UTC)" (scheduled at 2023-01-17 05:31:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:31:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:31:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:31:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:31:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:57.226424+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2023-01-17 05:31:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:57 UTC)" (scheduled at 2023-01-17 05:31:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:31:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 31, 27, 227247), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:31:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:31:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:31:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:38.735066+00:00 (in 9.999486 seconds) notificationworker stdout | 2023-01-17 05:31:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:38 UTC)" (scheduled at 2023-01-17 05:31:28.735066+00:00) notificationworker stdout | 2023-01-17 05:31:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 28, 736005), True, datetime.datetime(2023, 1, 17, 5, 31, 28, 736005), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:30,454 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:31:30,602 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:31:30,677 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:30,685 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:30,685 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:30,692 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:31:30,901 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:31:31,712 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:31:31,736 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:31:33,037 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:31:33,777 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:31:33,791 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:31:34,217 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:31:35,148 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:31:35,153 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:31:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:31:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:05.848675+00:00 (in 29.999528 seconds) gcworker stdout | 2023-01-17 05:31:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:05 UTC)" (scheduled at 2023-01-17 05:31:35.848675+00:00) gcworker stdout | 2023-01-17 05:31:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723895849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:31:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:31:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:31:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:31:35,930 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:31:35,932 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:31:35,935 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:31:36,092 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:31:36,847 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:31:36,869 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:31:37,625 [239] [DEBUG] [app] Starting request: urn:request:d88929aa-565d-42d2-9b78-78b163304d82 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:37,625 [244] [DEBUG] [app] Starting request: urn:request:b37d8d74-082f-4d84-8c32-dde641f3809b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:37,630 [232] [DEBUG] [app] Starting request: urn:request:22b471fc-5f26-4524-bb17-f90a35b54c8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:37,630 [240] [DEBUG] [app] Starting request: urn:request:9e3ae9bc-703b-4c54-8d65-4c568971cc17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:37,630 [232] [DEBUG] [app] Ending request: urn:request:22b471fc-5f26-4524-bb17-f90a35b54c8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:22b471fc-5f26-4524-bb17-f90a35b54c8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:31:37,631 [240] [DEBUG] [app] Ending request: urn:request:9e3ae9bc-703b-4c54-8d65-4c568971cc17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e3ae9bc-703b-4c54-8d65-4c568971cc17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:31:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:37,635 [243] [DEBUG] [app] Starting request: urn:request:4745b960-f6a2-443b-afdd-03fef5d56fc6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:37,636 [239] [DEBUG] [app] Starting request: urn:request:034f2f87-50f1-4289-9da7-a2c362dba5b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:37,636 [243] [DEBUG] [app] Ending request: urn:request:4745b960-f6a2-443b-afdd-03fef5d56fc6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4745b960-f6a2-443b-afdd-03fef5d56fc6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:37,636 [239] [DEBUG] [app] Ending request: urn:request:034f2f87-50f1-4289-9da7-a2c362dba5b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:034f2f87-50f1-4289-9da7-a2c362dba5b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:31:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:37,647 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:37,647 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:37,653 [239] [DEBUG] [app] Ending request: urn:request:d88929aa-565d-42d2-9b78-78b163304d82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d88929aa-565d-42d2-9b78-78b163304d82', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:37,653 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:37,653 [244] [DEBUG] [app] Ending request: urn:request:b37d8d74-082f-4d84-8c32-dde641f3809b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b37d8d74-082f-4d84-8c32-dde641f3809b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:37,653 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:31:37,653 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:31:37,653 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) storagereplication stdout | 2023-01-17 05:31:38,085 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:31:38,286 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:31:38,288 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:31:38,291 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:31:38,293 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:31:38,296 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:31:38,298 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:31:38,301 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:31:38,304 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:31:38,307 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 05:31:38,479 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:31:38,481 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:31:38,484 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:31:38,486 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:31:38,489 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:31:38,702 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:31:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:43.935629+00:00 (in 5.200185 seconds) notificationworker stdout | 2023-01-17 05:31:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:48 UTC)" (scheduled at 2023-01-17 05:31:38.735066+00:00) notificationworker stdout | 2023-01-17 05:31:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 38, 735852), True, datetime.datetime(2023, 1, 17, 5, 31, 38, 735852), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:31:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:31:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:48.738458+00:00 (in 9.707196 seconds) repositorygcworker stdout | 2023-01-17 05:31:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:39 UTC)" (scheduled at 2023-01-17 05:31:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:31:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:31:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 39, 31652), True, datetime.datetime(2023, 1, 17, 5, 31, 39, 31652), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:31:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:31:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:31:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:52.837365+00:00 (in 13.606902 seconds) securityworker stdout | 2023-01-17 05:31:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:09 UTC)" (scheduled at 2023-01-17 05:31:39.230038+00:00) securityworker stdout | 2023-01-17 05:31:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:31:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:31:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:39,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:39,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:31:39,240 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:31:39,240 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:31:39,240 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:31:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:31:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:31:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:20.242198+00:00 (in 40.907856 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:31:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:39 UTC)" (scheduled at 2023-01-17 05:31:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:31:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 31, 39, 334747), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:31:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 39, 334747), True, datetime.datetime(2023, 1, 17, 5, 31, 39, 334747), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:31:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 39, 334747), True, datetime.datetime(2023, 1, 17, 5, 31, 39, 334747), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 31, 39, 334747), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:31:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:31:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:31:39,453 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:31:39,473 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:31:39,477 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:31:39,533 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:31:40,701 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:40,710 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:40,710 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:40,716 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:48.735066+00:00 (in 4.798945 seconds) notificationworker stdout | 2023-01-17 05:31:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:43 UTC)" (scheduled at 2023-01-17 05:31:43.935629+00:00) notificationworker stdout | 2023-01-17 05:31:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:31:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:31:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:31:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:48.830274+00:00 (in 4.188311 seconds) exportactionlogsworker stdout | 2023-01-17 05:31:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:44 UTC)" (scheduled at 2023-01-17 05:31:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:31:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 31, 44, 642335), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:31:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 44, 642335), True, datetime.datetime(2023, 1, 17, 5, 31, 44, 642335), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:31:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 44, 642335), True, datetime.datetime(2023, 1, 17, 5, 31, 44, 642335), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 31, 44, 642335), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:31:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:31:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:31:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:31:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:59.337413+00:00 (in 11.404451 seconds) namespacegcworker stdout | 2023-01-17 05:31:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:47 UTC)" (scheduled at 2023-01-17 05:31:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:31:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:31:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:58.735066+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 05:31:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:58 UTC)" (scheduled at 2023-01-17 05:31:48.735066+00:00) notificationworker stdout | 2023-01-17 05:31:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 48, 735966), True, datetime.datetime(2023, 1, 17, 5, 31, 48, 735966), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:31:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:31:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:04.738062+00:00 (in 15.999149 seconds) repositorygcworker stdout | 2023-01-17 05:31:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:48 UTC)" (scheduled at 2023-01-17 05:31:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:31:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:31:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:31:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:31:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:24.641816+00:00 (in 35.811136 seconds) exportactionlogsworker stdout | 2023-01-17 05:31:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:48 UTC)" (scheduled at 2023-01-17 05:31:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:31:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:31:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 48, 831056), True, datetime.datetime(2023, 1, 17, 5, 31, 48, 831056), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:31:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:31:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:31:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:48 UTC)" executed successfully builder stdout | 2023-01-17 05:31:50,726 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:50,735 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:50,735 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:50,741 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:52,626 [239] [DEBUG] [app] Starting request: urn:request:023ea3e6-adc8-4938-9542-a56735fc0637 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,626 [241] [DEBUG] [app] Starting request: urn:request:21895b71-282d-4ea5-94fa-4337ea893692 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:52,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:52,631 [242] [DEBUG] [app] Starting request: urn:request:886b4b3f-f3cb-4ebd-ac9e-489424796f7e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,631 [242] [DEBUG] [app] Ending request: urn:request:886b4b3f-f3cb-4ebd-ac9e-489424796f7e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:886b4b3f-f3cb-4ebd-ac9e-489424796f7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:31:52,632 [236] [DEBUG] [app] Starting request: urn:request:42bdf1ec-5444-42a2-b483-6e4985cf4a3e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:31:52,632 [236] [DEBUG] [app] Ending request: urn:request:42bdf1ec-5444-42a2-b483-6e4985cf4a3e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:42bdf1ec-5444-42a2-b483-6e4985cf4a3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,633 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:31:52,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:52,636 [244] [DEBUG] [app] Starting request: urn:request:22ef0a97-5b76-4540-a78d-00b15587273c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:52,636 [244] [DEBUG] [app] Ending request: urn:request:22ef0a97-5b76-4540-a78d-00b15587273c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:22ef0a97-5b76-4540-a78d-00b15587273c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:52,637 [243] [DEBUG] [app] Starting request: urn:request:c8507747-8858-45b4-aa7b-bbf55a86a630 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,637 [243] [DEBUG] [app] Ending request: urn:request:c8507747-8858-45b4-aa7b-bbf55a86a630 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8507747-8858-45b4-aa7b-bbf55a86a630', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:52,647 [241] [DEBUG] [app] Ending request: urn:request:21895b71-282d-4ea5-94fa-4337ea893692 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21895b71-282d-4ea5-94fa-4337ea893692', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:31:52,648 [239] [DEBUG] [app] Ending request: urn:request:023ea3e6-adc8-4938-9542-a56735fc0637 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:023ea3e6-adc8-4938-9542-a56735fc0637', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:31:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:09.230038+00:00 (in 16.392233 seconds) securityworker stdout | 2023-01-17 05:31:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:22 UTC)" (scheduled at 2023-01-17 05:31:52.837365+00:00) securityworker stdout | 2023-01-17 05:31:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:52,852 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:52,852 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:31:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:31:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:27.226424+00:00 (in 29.999540 seconds) buildlogsarchiver stdout | 2023-01-17 05:31:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:27 UTC)" (scheduled at 2023-01-17 05:31:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:31:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 31, 57, 227231), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:31:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:31:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:31:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:08.735066+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:31:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:08 UTC)" (scheduled at 2023-01-17 05:31:58.735066+00:00) notificationworker stdout | 2023-01-17 05:31:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 58, 735900), True, datetime.datetime(2023, 1, 17, 5, 31, 58, 735900), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:31:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:31:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:47.932490+00:00 (in 48.594599 seconds) namespacegcworker stdout | 2023-01-17 05:31:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:59 UTC)" (scheduled at 2023-01-17 05:31:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:31:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:31:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 59, 338302), True, datetime.datetime(2023, 1, 17, 5, 31, 59, 338302), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:31:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:31:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:31:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:00,480 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:32:00,637 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:32:00,752 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:00,761 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:00,761 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:00,766 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:32:00,937 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:32:01,725 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:32:01,767 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:32:03,073 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:32:03,787 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:32:03,826 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:32:04,246 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositorygcworker stdout | 2023-01-17 05:32:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:32:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:39.030814+00:00 (in 34.292253 seconds) repositorygcworker stdout | 2023-01-17 05:32:04,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:37:04 UTC)" (scheduled at 2023-01-17 05:32:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:32:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 32, 4, 739158), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:32:04,747 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 32, 4, 739158), True, datetime.datetime(2023, 1, 17, 5, 32, 4, 739158), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:32:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 32, 4, 739158), True, datetime.datetime(2023, 1, 17, 5, 32, 4, 739158), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 32, 4, 739158), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:32:04,751 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:32:04,751 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:37:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:32:05,185 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:32:05,188 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:32:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:32:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:35.848675+00:00 (in 29.999486 seconds) gcworker stdout | 2023-01-17 05:32:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:35 UTC)" (scheduled at 2023-01-17 05:32:05.848675+00:00) gcworker stdout | 2023-01-17 05:32:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723925849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:32:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:32:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:32:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:32:05,937 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:32:05,939 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:32:05,941 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:32:06,127 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:32:06,878 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:32:06,881 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:32:07,625 [243] [DEBUG] [app] Starting request: urn:request:3ba160f6-2c45-4ca0-a434-4c07a1aaad75 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:07,625 [241] [DEBUG] [app] Starting request: urn:request:f082cf50-ad2a-4952-aa37-b3fe74a6d892 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:07,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:07,630 [232] [DEBUG] [app] Starting request: urn:request:ad2f14e8-ce63-4a0e-ad01-f2f8b6c89104 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,631 [242] [DEBUG] [app] Starting request: urn:request:0d5bdc1d-364b-403a-8934-df1d3a4f365b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,631 [232] [DEBUG] [app] Ending request: urn:request:ad2f14e8-ce63-4a0e-ad01-f2f8b6c89104 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad2f14e8-ce63-4a0e-ad01-f2f8b6c89104', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,631 [242] [DEBUG] [app] Ending request: urn:request:0d5bdc1d-364b-403a-8934-df1d3a4f365b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d5bdc1d-364b-403a-8934-df1d3a4f365b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:32:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:32:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:07,635 [244] [DEBUG] [app] Starting request: urn:request:a5db8b05-ad8d-4e0b-8a14-1ff5ba50dbc5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:07,636 [241] [DEBUG] [app] Starting request: urn:request:af81d225-f45c-40d6-b5c6-19e31005d068 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:07,636 [244] [DEBUG] [app] Ending request: urn:request:a5db8b05-ad8d-4e0b-8a14-1ff5ba50dbc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a5db8b05-ad8d-4e0b-8a14-1ff5ba50dbc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:07,636 [241] [DEBUG] [app] Ending request: urn:request:af81d225-f45c-40d6-b5c6-19e31005d068 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af81d225-f45c-40d6-b5c6-19e31005d068', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:07,647 [241] [DEBUG] [app] Ending request: urn:request:f082cf50-ad2a-4952-aa37-b3fe74a6d892 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f082cf50-ad2a-4952-aa37-b3fe74a6d892', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:07,647 [243] [DEBUG] [app] Ending request: urn:request:3ba160f6-2c45-4ca0-a434-4c07a1aaad75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ba160f6-2c45-4ca0-a434-4c07a1aaad75', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:32:07,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:32:08,116 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:32:08,293 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:32:08,299 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:32:08,301 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:32:08,303 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:32:08,306 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:32:08,309 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:32:08,311 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:32:08,314 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:32:08,316 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:32:08,486 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:32:08,488 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:32:08,491 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:32:08,494 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:32:08,496 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:32:08,707 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:32:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:18.735066+00:00 (in 9.999649 seconds) notificationworker stdout | 2023-01-17 05:32:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:18 UTC)" (scheduled at 2023-01-17 05:32:08.735066+00:00) notificationworker stdout | 2023-01-17 05:32:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 8, 735779), True, datetime.datetime(2023, 1, 17, 5, 32, 8, 735779), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:22.837365+00:00 (in 13.606869 seconds) securityworker stdout | 2023-01-17 05:32:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:39 UTC)" (scheduled at 2023-01-17 05:32:09.230038+00:00) securityworker stdout | 2023-01-17 05:32:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:32:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:32:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:32:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:32:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:32:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:32:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:32:09,471 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:32:09,480 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:32:09,512 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:32:09,538 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:32:10,777 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:10,787 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:10,787 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:10,793 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:28.735066+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:32:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:28 UTC)" (scheduled at 2023-01-17 05:32:18.735066+00:00) notificationworker stdout | 2023-01-17 05:32:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 18, 735946), True, datetime.datetime(2023, 1, 17, 5, 32, 18, 735946), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:32:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:32:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:25.334138+00:00 (in 5.091433 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:32:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:20 UTC)" (scheduled at 2023-01-17 05:32:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:32:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:32:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 20, 243103), True, datetime.datetime(2023, 1, 17, 5, 32, 20, 243103), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:32:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:32:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:32:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:20 UTC)" executed successfully builder stdout | 2023-01-17 05:32:20,803 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:20,811 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:20,811 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:20,817 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:22,625 [244] [DEBUG] [app] Starting request: urn:request:726edac8-5ec2-48a6-82b7-e137026f5ad7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:22,625 [239] [DEBUG] [app] Starting request: urn:request:676fa0f5-21bc-4636-bb2b-220b83f1b7c2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:22,630 [232] [DEBUG] [app] Starting request: urn:request:cefcd925-5153-4529-adb1-bb90eb010607 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,630 [242] [DEBUG] [app] Starting request: urn:request:9064bced-0550-4a7d-bfa1-4d8bd7b1c67a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,630 [232] [DEBUG] [app] Ending request: urn:request:cefcd925-5153-4529-adb1-bb90eb010607 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cefcd925-5153-4529-adb1-bb90eb010607', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:32:22,631 [242] [DEBUG] [app] Ending request: urn:request:9064bced-0550-4a7d-bfa1-4d8bd7b1c67a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9064bced-0550-4a7d-bfa1-4d8bd7b1c67a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:32:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:22,635 [243] [DEBUG] [app] Starting request: urn:request:923c505d-438b-4b66-8892-f775d0bdfec5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:22,635 [241] [DEBUG] [app] Starting request: urn:request:73253fa7-5e58-4e6c-814c-41c30caffbca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:22,635 [243] [DEBUG] [app] Ending request: urn:request:923c505d-438b-4b66-8892-f775d0bdfec5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:923c505d-438b-4b66-8892-f775d0bdfec5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:22,636 [241] [DEBUG] [app] Ending request: urn:request:73253fa7-5e58-4e6c-814c-41c30caffbca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73253fa7-5e58-4e6c-814c-41c30caffbca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:32:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:22,646 [239] [DEBUG] [app] Ending request: urn:request:676fa0f5-21bc-4636-bb2b-220b83f1b7c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:676fa0f5-21bc-4636-bb2b-220b83f1b7c2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:22,646 [244] [DEBUG] [app] Ending request: urn:request:726edac8-5ec2-48a6-82b7-e137026f5ad7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:726edac8-5ec2-48a6-82b7-e137026f5ad7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:22,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:22,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:32:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:32:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:32:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:39.230038+00:00 (in 16.392194 seconds) securityworker stdout | 2023-01-17 05:32:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:52 UTC)" (scheduled at 2023-01-17 05:32:22.837365+00:00) securityworker stdout | 2023-01-17 05:32:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:32:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:32:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:48.830274+00:00 (in 24.188012 seconds) exportactionlogsworker stdout | 2023-01-17 05:32:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:24 UTC)" (scheduled at 2023-01-17 05:32:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:32:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:32:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:32:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:32:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:20.242198+00:00 (in 54.907600 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:32:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:25 UTC)" (scheduled at 2023-01-17 05:32:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:32:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:32:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:32:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:32:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:57.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 05:32:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:57 UTC)" (scheduled at 2023-01-17 05:32:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:32:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 32, 27, 227212), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:32:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:32:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:32:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:38.735066+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:32:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:38 UTC)" (scheduled at 2023-01-17 05:32:28.735066+00:00) notificationworker stdout | 2023-01-17 05:32:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 28, 735974), True, datetime.datetime(2023, 1, 17, 5, 32, 28, 735974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:30,505 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:32:30,653 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:32:30,825 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:30,834 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:30,834 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:30,839 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:32:30,953 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:32:31,761 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:32:31,783 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:32:33,092 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:32:33,823 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:32:33,860 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:32:34,277 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:32:35,213 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:32:35,223 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:32:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:32:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:05.848675+00:00 (in 29.999516 seconds) gcworker stdout | 2023-01-17 05:32:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:05 UTC)" (scheduled at 2023-01-17 05:32:35.848675+00:00) gcworker stdout | 2023-01-17 05:32:35,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:32:35,856 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723955855, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:32:35,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:32:35,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:32:35,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:32:35,943 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:32:35,946 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:32:35,949 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:32:36,161 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:32:36,885 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:32:36,916 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:32:37,625 [243] [DEBUG] [app] Starting request: urn:request:81949743-c777-40ad-9fe2-3f539dfffd86 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:37,626 [239] [DEBUG] [app] Starting request: urn:request:84fd8a0d-7500-41ab-80e1-f73ab6eb457d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:37,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:37,631 [232] [DEBUG] [app] Starting request: urn:request:cee62f7f-6b97-453a-91c3-3de58b895e0c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,632 [237] [DEBUG] [app] Starting request: urn:request:0213003f-c209-4c13-af41-2dfb8d579807 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,632 [232] [DEBUG] [app] Ending request: urn:request:cee62f7f-6b97-453a-91c3-3de58b895e0c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cee62f7f-6b97-453a-91c3-3de58b895e0c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,632 [237] [DEBUG] [app] Ending request: urn:request:0213003f-c209-4c13-af41-2dfb8d579807 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0213003f-c209-4c13-af41-2dfb8d579807', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:32:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:32:37,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:37,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:37,637 [244] [DEBUG] [app] Starting request: urn:request:6ddda7bc-ebbd-44a5-9236-4129ff307423 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:37,637 [244] [DEBUG] [app] Ending request: urn:request:6ddda7bc-ebbd-44a5-9236-4129ff307423 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ddda7bc-ebbd-44a5-9236-4129ff307423', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:37,637 [241] [DEBUG] [app] Starting request: urn:request:f3434543-1e70-4aa3-aa76-7d792f888fbf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:37,638 [241] [DEBUG] [app] Ending request: urn:request:f3434543-1e70-4aa3-aa76-7d792f888fbf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3434543-1e70-4aa3-aa76-7d792f888fbf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:32:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:37,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:37,648 [243] [DEBUG] [app] Ending request: urn:request:81949743-c777-40ad-9fe2-3f539dfffd86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81949743-c777-40ad-9fe2-3f539dfffd86', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:37,648 [239] [DEBUG] [app] Ending request: urn:request:84fd8a0d-7500-41ab-80e1-f73ab6eb457d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84fd8a0d-7500-41ab-80e1-f73ab6eb457d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:32:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) storagereplication stdout | 2023-01-17 05:32:38,146 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:32:38,300 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:32:38,305 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:32:38,308 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:32:38,313 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:32:38,315 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:32:38,318 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:32:38,320 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:32:38,322 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:32:38,324 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:32:38,493 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:32:38,495 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:32:38,497 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:32:38,500 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:32:38,503 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:32:38,718 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:32:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:43.935629+00:00 (in 5.200131 seconds) notificationworker stdout | 2023-01-17 05:32:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:48 UTC)" (scheduled at 2023-01-17 05:32:38.735066+00:00) notificationworker stdout | 2023-01-17 05:32:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 38, 735898), True, datetime.datetime(2023, 1, 17, 5, 32, 38, 735898), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:32:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:32:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:48.738458+00:00 (in 9.707222 seconds) repositorygcworker stdout | 2023-01-17 05:32:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:39 UTC)" (scheduled at 2023-01-17 05:32:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:32:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:32:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 39, 31682), True, datetime.datetime(2023, 1, 17, 5, 32, 39, 31682), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:32:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:32:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:32:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:52.837365+00:00 (in 13.606882 seconds) securityworker stdout | 2023-01-17 05:32:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:09 UTC)" (scheduled at 2023-01-17 05:32:39.230038+00:00) securityworker stdout | 2023-01-17 05:32:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:32:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:32:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:32:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:32:39,483 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:32:39,515 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:32:39,517 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:32:39,572 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:32:40,848 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:40,856 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:40,856 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:40,862 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:48.735066+00:00 (in 4.798933 seconds) notificationworker stdout | 2023-01-17 05:32:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:43 UTC)" (scheduled at 2023-01-17 05:32:43.935629+00:00) notificationworker stdout | 2023-01-17 05:32:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:32:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:32:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:32:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:59.337413+00:00 (in 11.404462 seconds) namespacegcworker stdout | 2023-01-17 05:32:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:47 UTC)" (scheduled at 2023-01-17 05:32:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:32:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:32:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:58.735066+00:00 (in 9.999499 seconds) notificationworker stdout | 2023-01-17 05:32:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:58 UTC)" (scheduled at 2023-01-17 05:32:48.735066+00:00) notificationworker stdout | 2023-01-17 05:32:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 48, 735950), True, datetime.datetime(2023, 1, 17, 5, 32, 48, 735950), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:32:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:32:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:39.030814+00:00 (in 50.291907 seconds) repositorygcworker stdout | 2023-01-17 05:32:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:48 UTC)" (scheduled at 2023-01-17 05:32:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:32:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:32:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:32:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:32:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:24.641816+00:00 (in 35.811059 seconds) exportactionlogsworker stdout | 2023-01-17 05:32:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:48 UTC)" (scheduled at 2023-01-17 05:32:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:32:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:32:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 48, 831188), True, datetime.datetime(2023, 1, 17, 5, 32, 48, 831188), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:32:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:32:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:32:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:48 UTC)" executed successfully builder stdout | 2023-01-17 05:32:50,872 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:50,881 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:50,881 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:50,887 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:52,626 [241] [DEBUG] [app] Starting request: urn:request:50511024-40f1-4ef5-9810-8e6c6651986c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:52,626 [239] [DEBUG] [app] Starting request: urn:request:e9287101-b61f-474d-89af-8935efb9fb43 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:32:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:52,631 [242] [DEBUG] [app] Starting request: urn:request:d139b24a-19f1-43b5-99cd-7ec3e654f583 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:52,631 [242] [DEBUG] [app] Ending request: urn:request:d139b24a-19f1-43b5-99cd-7ec3e654f583 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d139b24a-19f1-43b5-99cd-7ec3e654f583', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:52,631 [234] [DEBUG] [app] Starting request: urn:request:22c3444d-a895-4287-b096-b8d8dae76d46 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:32:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:32:52,632 [234] [DEBUG] [app] Ending request: urn:request:22c3444d-a895-4287-b096-b8d8dae76d46 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:22c3444d-a895-4287-b096-b8d8dae76d46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:32:52,632 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:52,635 [239] [DEBUG] [app] Starting request: urn:request:7c90d430-c8ea-438d-8798-0d3add56ca60 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:52,636 [239] [DEBUG] [app] Ending request: urn:request:7c90d430-c8ea-438d-8798-0d3add56ca60 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c90d430-c8ea-438d-8798-0d3add56ca60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:52,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:52,637 [244] [DEBUG] [app] Starting request: urn:request:1a33c53f-53fc-4681-8be9-16df99b78594 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:52,638 [244] [DEBUG] [app] Ending request: urn:request:1a33c53f-53fc-4681-8be9-16df99b78594 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a33c53f-53fc-4681-8be9-16df99b78594', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:52,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:32:52,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:52,647 [241] [DEBUG] [app] Ending request: urn:request:50511024-40f1-4ef5-9810-8e6c6651986c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50511024-40f1-4ef5-9810-8e6c6651986c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:52,647 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:32:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:32:52,648 [239] [DEBUG] [app] Ending request: urn:request:e9287101-b61f-474d-89af-8935efb9fb43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e9287101-b61f-474d-89af-8935efb9fb43', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:52,649 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:32:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:32:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:09.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 05:32:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:22 UTC)" (scheduled at 2023-01-17 05:32:52.837365+00:00) securityworker stdout | 2023-01-17 05:32:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:32:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:32:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:27.226424+00:00 (in 29.999513 seconds) buildlogsarchiver stdout | 2023-01-17 05:32:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:27 UTC)" (scheduled at 2023-01-17 05:32:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:32:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 32, 57, 227284), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:32:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:32:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:32:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:08.735066+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:32:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:08 UTC)" (scheduled at 2023-01-17 05:32:58.735066+00:00) notificationworker stdout | 2023-01-17 05:32:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 58, 735990), True, datetime.datetime(2023, 1, 17, 5, 32, 58, 735990), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:32:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:32:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:47.932490+00:00 (in 48.594598 seconds) namespacegcworker stdout | 2023-01-17 05:32:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:59 UTC)" (scheduled at 2023-01-17 05:32:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:32:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:32:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 59, 338290), True, datetime.datetime(2023, 1, 17, 5, 32, 59, 338290), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:32:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:32:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:32:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:00,539 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:33:00,681 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:33:00,897 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:00,906 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:00,906 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:00,912 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:33:00,988 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:33:01,782 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:33:01,818 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:33:03,102 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:33:03,828 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:33:03,896 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:33:04,313 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:33:05,237 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:33:05,257 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:33:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:33:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:35.848675+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 05:33:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:35 UTC)" (scheduled at 2023-01-17 05:33:05.848675+00:00) gcworker stdout | 2023-01-17 05:33:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723985849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:33:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:33:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:33:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:33:05,951 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:33:05,954 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:33:05,956 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:33:06,168 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:33:06,918 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:33:06,939 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:33:07,625 [243] [DEBUG] [app] Starting request: urn:request:ba25f98f-8b9a-4ead-aab1-a51764e4d964 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:07,625 [239] [DEBUG] [app] Starting request: urn:request:26e65d42-742e-4e79-821a-6f60ff896555 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:07,630 [237] [DEBUG] [app] Starting request: urn:request:14d78c7d-8d91-49d2-9b78-b5c205cc766c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:07,631 [236] [DEBUG] [app] Starting request: urn:request:78c76074-cb77-475c-95a6-61619f67a08e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:07,631 [237] [DEBUG] [app] Ending request: urn:request:14d78c7d-8d91-49d2-9b78-b5c205cc766c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:14d78c7d-8d91-49d2-9b78-b5c205cc766c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:33:07,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:33:07,631 [236] [DEBUG] [app] Ending request: urn:request:78c76074-cb77-475c-95a6-61619f67a08e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78c76074-cb77-475c-95a6-61619f67a08e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:33:07,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:33:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:07,636 [244] [DEBUG] [app] Starting request: urn:request:d99b3679-0901-4c28-9482-d12c61e001d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:07,636 [241] [DEBUG] [app] Starting request: urn:request:7326faa9-8b6b-482e-ac37-28ed2cd9b03a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:07,636 [244] [DEBUG] [app] Ending request: urn:request:d99b3679-0901-4c28-9482-d12c61e001d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d99b3679-0901-4c28-9482-d12c61e001d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:33:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:07,636 [241] [DEBUG] [app] Ending request: urn:request:7326faa9-8b6b-482e-ac37-28ed2cd9b03a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7326faa9-8b6b-482e-ac37-28ed2cd9b03a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:07,647 [243] [DEBUG] [app] Ending request: urn:request:ba25f98f-8b9a-4ead-aab1-a51764e4d964 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba25f98f-8b9a-4ead-aab1-a51764e4d964', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:07,647 [239] [DEBUG] [app] Ending request: urn:request:26e65d42-742e-4e79-821a-6f60ff896555 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26e65d42-742e-4e79-821a-6f60ff896555', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:33:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) storagereplication stdout | 2023-01-17 05:33:08,182 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:33:08,307 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:33:08,315 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:33:08,317 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:33:08,321 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:33:08,325 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:33:08,328 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:33:08,330 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:33:08,333 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:33:08,335 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:33:08,500 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:33:08,502 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:33:08,505 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:33:08,507 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:33:08,509 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:33:08,731 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:33:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:18.735066+00:00 (in 9.999652 seconds) notificationworker stdout | 2023-01-17 05:33:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:18 UTC)" (scheduled at 2023-01-17 05:33:08.735066+00:00) notificationworker stdout | 2023-01-17 05:33:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 8, 735754), True, datetime.datetime(2023, 1, 17, 5, 33, 8, 735754), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:22.837365+00:00 (in 13.606850 seconds) securityworker stdout | 2023-01-17 05:33:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:39 UTC)" (scheduled at 2023-01-17 05:33:09.230038+00:00) securityworker stdout | 2023-01-17 05:33:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:33:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:33:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:33:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:33:09,488 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:33:09,551 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:33:09,553 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:33:09,588 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:33:10,919 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:10,928 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:10,928 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:10,933 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:28.735066+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:33:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:28 UTC)" (scheduled at 2023-01-17 05:33:18.735066+00:00) notificationworker stdout | 2023-01-17 05:33:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 18, 735964), True, datetime.datetime(2023, 1, 17, 5, 33, 18, 735964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:33:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:33:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:25.334138+00:00 (in 5.091477 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:33:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:20 UTC)" (scheduled at 2023-01-17 05:33:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:33:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:33:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 20, 243094), True, datetime.datetime(2023, 1, 17, 5, 33, 20, 243094), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:33:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:33:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:33:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:20 UTC)" executed successfully builder stdout | 2023-01-17 05:33:20,944 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:20,952 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:20,952 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:20,958 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:22,625 [241] [DEBUG] [app] Starting request: urn:request:33a81247-38dc-467c-84a0-1e05c7f6ed01 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:22,626 [239] [DEBUG] [app] Starting request: urn:request:b36e2451-63ea-47e8-bc9d-b1b8cdc525df (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:22,630 [232] [DEBUG] [app] Starting request: urn:request:5d035389-0769-44b2-9ef8-fb36cc1ab7da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:22,630 [232] [DEBUG] [app] Ending request: urn:request:5d035389-0769-44b2-9ef8-fb36cc1ab7da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d035389-0769-44b2-9ef8-fb36cc1ab7da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:22,631 [240] [DEBUG] [app] Starting request: urn:request:a4d6689f-40b3-4a9b-a24d-126cff7f1870 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:33:22,631 [240] [DEBUG] [app] Ending request: urn:request:a4d6689f-40b3-4a9b-a24d-126cff7f1870 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4d6689f-40b3-4a9b-a24d-126cff7f1870', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:33:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:22,635 [244] [DEBUG] [app] Starting request: urn:request:fc81f8b1-9ac9-4a41-b36f-78a6b4744bc1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:22,635 [244] [DEBUG] [app] Ending request: urn:request:fc81f8b1-9ac9-4a41-b36f-78a6b4744bc1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc81f8b1-9ac9-4a41-b36f-78a6b4744bc1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:22,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:22,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:22,636 [244] [DEBUG] [app] Starting request: urn:request:ca8e75e3-4be3-4bcd-a904-e5a51b661b66 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:22,637 [244] [DEBUG] [app] Ending request: urn:request:ca8e75e3-4be3-4bcd-a904-e5a51b661b66 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca8e75e3-4be3-4bcd-a904-e5a51b661b66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.000) gunicorn-web stdout | 2023-01-17 05:33:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:22,646 [239] [DEBUG] [app] Ending request: urn:request:b36e2451-63ea-47e8-bc9d-b1b8cdc525df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b36e2451-63ea-47e8-bc9d-b1b8cdc525df', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:22,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.021) gunicorn-web stdout | 2023-01-17 05:33:22,647 [241] [DEBUG] [app] Ending request: urn:request:33a81247-38dc-467c-84a0-1e05c7f6ed01 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33a81247-38dc-467c-84a0-1e05c7f6ed01', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:22,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:33:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:33:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:39.230038+00:00 (in 16.392193 seconds) securityworker stdout | 2023-01-17 05:33:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:52 UTC)" (scheduled at 2023-01-17 05:33:22.837365+00:00) securityworker stdout | 2023-01-17 05:33:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:33:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:33:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:48.830274+00:00 (in 24.187979 seconds) exportactionlogsworker stdout | 2023-01-17 05:33:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:24 UTC)" (scheduled at 2023-01-17 05:33:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:33:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:33:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:33:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:33:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:20.242198+00:00 (in 54.907569 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:33:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:25 UTC)" (scheduled at 2023-01-17 05:33:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:33:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:33:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:33:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:33:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:57.226424+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 05:33:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:57 UTC)" (scheduled at 2023-01-17 05:33:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:33:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 33, 27, 227304), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:33:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:33:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:33:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:38.735066+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:33:28,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:38 UTC)" (scheduled at 2023-01-17 05:33:28.735066+00:00) notificationworker stdout | 2023-01-17 05:33:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 28, 736199), True, datetime.datetime(2023, 1, 17, 5, 33, 28, 736199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:30,556 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:33:30,717 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:33:30,968 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:30,977 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:30,977 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:30,983 [60] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:33:31,024 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:33:31,817 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:33:31,828 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:33:33,130 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:33:33,839 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:33:33,914 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:33:34,348 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:33:35,243 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:33:35,289 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:33:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:33:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:05.848675+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 05:33:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:05 UTC)" (scheduled at 2023-01-17 05:33:35.848675+00:00) gcworker stdout | 2023-01-17 05:33:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724015849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:33:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:33:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:33:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:33:35,959 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:33:35,961 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:33:35,963 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:33:36,173 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:33:36,940 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:33:36,961 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:33:37,626 [244] [DEBUG] [app] Starting request: urn:request:dcc3ae4e-59c7-49ec-91a8-116cab029678 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:37,626 [243] [DEBUG] [app] Starting request: urn:request:b8d10704-59a9-498b-95dc-ccd557078832 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:37,631 [242] [DEBUG] [app] Starting request: urn:request:f4fae8e1-3786-4082-b00c-88031c15b429 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:37,631 [240] [DEBUG] [app] Starting request: urn:request:3a678e2b-6e9e-4f65-ae76-e0743e4a9b07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:37,631 [242] [DEBUG] [app] Ending request: urn:request:f4fae8e1-3786-4082-b00c-88031c15b429 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4fae8e1-3786-4082-b00c-88031c15b429', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:33:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:33:37,632 [240] [DEBUG] [app] Ending request: urn:request:3a678e2b-6e9e-4f65-ae76-e0743e4a9b07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a678e2b-6e9e-4f65-ae76-e0743e4a9b07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:33:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:37,636 [239] [DEBUG] [app] Starting request: urn:request:588e5f2b-9340-4fb7-b33d-fd36eb6319a3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:37,636 [244] [DEBUG] [app] Starting request: urn:request:5a01515e-8c40-46ef-afac-7bbc107d02a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:37,636 [244] [DEBUG] [app] Ending request: urn:request:5a01515e-8c40-46ef-afac-7bbc107d02a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a01515e-8c40-46ef-afac-7bbc107d02a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:37,636 [239] [DEBUG] [app] Ending request: urn:request:588e5f2b-9340-4fb7-b33d-fd36eb6319a3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:588e5f2b-9340-4fb7-b33d-fd36eb6319a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:37,647 [244] [DEBUG] [app] Ending request: urn:request:dcc3ae4e-59c7-49ec-91a8-116cab029678 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcc3ae4e-59c7-49ec-91a8-116cab029678', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:37,647 [243] [DEBUG] [app] Ending request: urn:request:b8d10704-59a9-498b-95dc-ccd557078832 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8d10704-59a9-498b-95dc-ccd557078832', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:33:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:33:38,218 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:33:38,313 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:33:38,323 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:33:38,325 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:33:38,329 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:33:38,334 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:33:38,338 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:33:38,341 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:33:38,343 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:33:38,346 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:33:38,507 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:33:38,509 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:33:38,512 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:33:38,514 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:33:38,517 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:33:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:43.935629+00:00 (in 5.200181 seconds) notificationworker stdout | 2023-01-17 05:33:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:48 UTC)" (scheduled at 2023-01-17 05:33:38.735066+00:00) notificationworker stdout | 2023-01-17 05:33:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 38, 735882), True, datetime.datetime(2023, 1, 17, 5, 33, 38, 735882), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:38,736 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:33:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:33:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:33:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:48.738458+00:00 (in 9.707256 seconds) repositorygcworker stdout | 2023-01-17 05:33:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:39 UTC)" (scheduled at 2023-01-17 05:33:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:33:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:33:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 39, 31597), True, datetime.datetime(2023, 1, 17, 5, 33, 39, 31597), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:33:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:33:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:33:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:52.837365+00:00 (in 13.606819 seconds) securityworker stdout | 2023-01-17 05:33:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:09 UTC)" (scheduled at 2023-01-17 05:33:39.230038+00:00) securityworker stdout | 2023-01-17 05:33:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:33:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:33:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:33:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:33:39,524 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:33:39,572 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:33:39,575 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:33:39,623 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:33:40,993 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:41,007 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:41,007 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:41,013 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:48.735066+00:00 (in 4.798967 seconds) notificationworker stdout | 2023-01-17 05:33:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:43 UTC)" (scheduled at 2023-01-17 05:33:43.935629+00:00) notificationworker stdout | 2023-01-17 05:33:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:33:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:33:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:33:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:59.337413+00:00 (in 11.404434 seconds) namespacegcworker stdout | 2023-01-17 05:33:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:47 UTC)" (scheduled at 2023-01-17 05:33:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:33:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:33:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:56.935055+00:00 (in 8.199501 seconds) notificationworker stdout | 2023-01-17 05:33:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:58 UTC)" (scheduled at 2023-01-17 05:33:48.735066+00:00) notificationworker stdout | 2023-01-17 05:33:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 48, 735973), True, datetime.datetime(2023, 1, 17, 5, 33, 48, 735973), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:33:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:33:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:39.030814+00:00 (in 50.291923 seconds) repositorygcworker stdout | 2023-01-17 05:33:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:48 UTC)" (scheduled at 2023-01-17 05:33:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:33:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:33:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:33:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:33:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:24.641816+00:00 (in 35.811108 seconds) exportactionlogsworker stdout | 2023-01-17 05:33:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:48 UTC)" (scheduled at 2023-01-17 05:33:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:33:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:33:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 48, 831082), True, datetime.datetime(2023, 1, 17, 5, 33, 48, 831082), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:33:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:33:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:33:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:48 UTC)" executed successfully builder stdout | 2023-01-17 05:33:51,017 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:51,026 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:51,026 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:51,032 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:52,626 [239] [DEBUG] [app] Starting request: urn:request:e43a1e05-12eb-410f-9ed1-704851f067cb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:52,626 [241] [DEBUG] [app] Starting request: urn:request:33b50349-ceac-4f0d-b34a-4e78d3c46c80 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:33:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:52,631 [232] [DEBUG] [app] Starting request: urn:request:f5e9829f-6740-4951-85a3-ec6270a680d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,631 [240] [DEBUG] [app] Starting request: urn:request:b8e2afda-19e6-47c2-9b4c-d35cd8d6d0ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,632 [232] [DEBUG] [app] Ending request: urn:request:f5e9829f-6740-4951-85a3-ec6270a680d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5e9829f-6740-4951-85a3-ec6270a680d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,632 [240] [DEBUG] [app] Ending request: urn:request:b8e2afda-19e6-47c2-9b4c-d35cd8d6d0ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b8e2afda-19e6-47c2-9b4c-d35cd8d6d0ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:33:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:33:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:52,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:52,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:52,637 [244] [DEBUG] [app] Starting request: urn:request:b9ecb1e4-c9bb-4336-b494-013b9e827ab8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:52,637 [244] [DEBUG] [app] Ending request: urn:request:b9ecb1e4-c9bb-4336-b494-013b9e827ab8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9ecb1e4-c9bb-4336-b494-013b9e827ab8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:52,637 [243] [DEBUG] [app] Starting request: urn:request:50ec15db-f1a7-47b2-8bc7-cff756dbdb5e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:52,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:52,638 [243] [DEBUG] [app] Ending request: urn:request:50ec15db-f1a7-47b2-8bc7-cff756dbdb5e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50ec15db-f1a7-47b2-8bc7-cff756dbdb5e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:33:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:52,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:52,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:52,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:52,649 [241] [DEBUG] [app] Ending request: urn:request:33b50349-ceac-4f0d-b34a-4e78d3c46c80 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33b50349-ceac-4f0d-b34a-4e78d3c46c80', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:52,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:52,649 [239] [DEBUG] [app] Ending request: urn:request:e43a1e05-12eb-410f-9ed1-704851f067cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e43a1e05-12eb-410f-9ed1-704851f067cb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:52,649 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 05:33:52,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:33:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:33:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:09.230038+00:00 (in 16.392150 seconds) securityworker stdout | 2023-01-17 05:33:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:22 UTC)" (scheduled at 2023-01-17 05:33:52.837365+00:00) securityworker stdout | 2023-01-17 05:33:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:58.735066+00:00 (in 1.799528 seconds) notificationworker stdout | 2023-01-17 05:33:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:38:56 UTC)" (scheduled at 2023-01-17 05:33:56.935055+00:00) notificationworker stdout | 2023-01-17 05:33:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 33, 56, 935906), 'notification/%']) notificationworker stdout | 2023-01-17 05:33:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 33, 56, 935906), True, datetime.datetime(2023, 1, 17, 5, 33, 56, 935906), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:33:56,946 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 33, 56, 935906), True, datetime.datetime(2023, 1, 17, 5, 33, 56, 935906), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 33, 56, 935906), 'notification/%']) notificationworker stdout | 2023-01-17 05:33:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:56,948 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:38:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:33:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:33:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:27.226424+00:00 (in 29.999508 seconds) buildlogsarchiver stdout | 2023-01-17 05:33:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:27 UTC)" (scheduled at 2023-01-17 05:33:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:33:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 33, 57, 227267), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:33:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:33:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:33:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:08.735066+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 05:33:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:08 UTC)" (scheduled at 2023-01-17 05:33:58.735066+00:00) notificationworker stdout | 2023-01-17 05:33:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 58, 735968), True, datetime.datetime(2023, 1, 17, 5, 33, 58, 735968), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:33:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:33:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:15.931994+00:00 (in 16.593974 seconds) namespacegcworker stdout | 2023-01-17 05:33:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:59 UTC)" (scheduled at 2023-01-17 05:33:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:33:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:33:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 59, 338424), True, datetime.datetime(2023, 1, 17, 5, 33, 59, 338424), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:33:59,355 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:33:59,355 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:33:59,355 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:00,588 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:34:00,753 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:34:01,038 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue servicekey stdout | 2023-01-17 05:34:01,043 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:34:01,052 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:01,052 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:01,058 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:34:01,853 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:34:01,863 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:34:03,145 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:34:03,875 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:34:03,934 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:34:04,382 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:34:05,279 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:34:05,306 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:34:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:34:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:35.848675+00:00 (in 29.999519 seconds) gcworker stdout | 2023-01-17 05:34:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:35 UTC)" (scheduled at 2023-01-17 05:34:05.848675+00:00) gcworker stdout | 2023-01-17 05:34:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724045849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:34:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:34:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:34:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:34:05,967 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:34:05,969 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:34:05,971 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:34:06,208 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:34:06,975 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:34:06,995 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:34:07,625 [244] [DEBUG] [app] Starting request: urn:request:130317be-6bf7-48f7-ad2e-864995c0cc2d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:07,625 [239] [DEBUG] [app] Starting request: urn:request:59611a02-f5f2-4b0c-8d80-10573f3be0cf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:07,630 [240] [DEBUG] [app] Starting request: urn:request:1c469af3-62cd-4585-b0fa-c3b1bdd82dbe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:07,630 [242] [DEBUG] [app] Starting request: urn:request:7b763412-eeb5-4f6e-a50a-37293e292a0b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:07,631 [240] [DEBUG] [app] Ending request: urn:request:1c469af3-62cd-4585-b0fa-c3b1bdd82dbe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c469af3-62cd-4585-b0fa-c3b1bdd82dbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:34:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:34:07,631 [242] [DEBUG] [app] Ending request: urn:request:7b763412-eeb5-4f6e-a50a-37293e292a0b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b763412-eeb5-4f6e-a50a-37293e292a0b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:34:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:34:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:07,635 [243] [DEBUG] [app] Starting request: urn:request:21a1ce49-bb47-46f2-9d4c-68b270c20872 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:07,635 [239] [DEBUG] [app] Starting request: urn:request:8c03fc7c-6a28-4723-ae6a-c2a05a87c0fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:07,636 [243] [DEBUG] [app] Ending request: urn:request:21a1ce49-bb47-46f2-9d4c-68b270c20872 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21a1ce49-bb47-46f2-9d4c-68b270c20872', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:07,636 [239] [DEBUG] [app] Ending request: urn:request:8c03fc7c-6a28-4723-ae6a-c2a05a87c0fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c03fc7c-6a28-4723-ae6a-c2a05a87c0fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:34:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:07,647 [239] [DEBUG] [app] Ending request: urn:request:59611a02-f5f2-4b0c-8d80-10573f3be0cf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:59611a02-f5f2-4b0c-8d80-10573f3be0cf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:34:07,647 [244] [DEBUG] [app] Ending request: urn:request:130317be-6bf7-48f7-ad2e-864995c0cc2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:130317be-6bf7-48f7-ad2e-864995c0cc2d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) storagereplication stdout | 2023-01-17 05:34:08,241 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:34:08,319 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:34:08,333 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:34:08,335 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:34:08,338 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:34:08,340 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:34:08,347 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:34:08,350 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:34:08,352 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:34:08,354 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:34:08,514 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:34:08,517 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:34:08,519 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:34:08,521 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:34:08,523 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:34:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:18.735066+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:34:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:18 UTC)" (scheduled at 2023-01-17 05:34:08.735066+00:00) notificationworker stdout | 2023-01-17 05:34:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 8, 735926), True, datetime.datetime(2023, 1, 17, 5, 34, 8, 735926), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:08,743 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 05:34:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:22.837365+00:00 (in 13.606859 seconds) securityworker stdout | 2023-01-17 05:34:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:39 UTC)" (scheduled at 2023-01-17 05:34:09.230038+00:00) securityworker stdout | 2023-01-17 05:34:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:34:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:34:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:34:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:34:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:34:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:34:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:34:09,559 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:34:09,584 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:34:09,608 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:34:09,658 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:34:11,059 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:11,068 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:11,069 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:11,074 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:34:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:34:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:47.932490+00:00 (in 32.000026 seconds) namespacegcworker stdout | 2023-01-17 05:34:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:39:15 UTC)" (scheduled at 2023-01-17 05:34:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:34:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 34, 15, 932827), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:34:15,939 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 34, 15, 932827), True, datetime.datetime(2023, 1, 17, 5, 34, 15, 932827), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:34:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 34, 15, 932827), True, datetime.datetime(2023, 1, 17, 5, 34, 15, 932827), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 34, 15, 932827), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:34:15,943 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:34:15,943 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:39:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:28.735066+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:34:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:28 UTC)" (scheduled at 2023-01-17 05:34:18.735066+00:00) notificationworker stdout | 2023-01-17 05:34:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 18, 735917), True, datetime.datetime(2023, 1, 17, 5, 34, 18, 735917), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:34:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:34:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:25.334138+00:00 (in 5.091440 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:34:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:20 UTC)" (scheduled at 2023-01-17 05:34:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:34:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:34:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 20, 243115), True, datetime.datetime(2023, 1, 17, 5, 34, 20, 243115), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:34:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:34:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:34:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:20 UTC)" executed successfully builder stdout | 2023-01-17 05:34:21,084 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:21,093 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:21,093 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:21,099 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:22,625 [241] [DEBUG] [app] Starting request: urn:request:433322b4-87f7-496b-9074-7bb959f26a5d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:22,626 [239] [DEBUG] [app] Starting request: urn:request:490ad6c0-09b2-4f01-832b-7af2e304c6b2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:22,630 [232] [DEBUG] [app] Starting request: urn:request:e8b78d71-856d-4699-a2b5-fc4dee1e26fc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,631 [242] [DEBUG] [app] Starting request: urn:request:917ec1ae-9230-4d79-b3e5-c25702742090 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,631 [232] [DEBUG] [app] Ending request: urn:request:e8b78d71-856d-4699-a2b5-fc4dee1e26fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e8b78d71-856d-4699-a2b5-fc4dee1e26fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:34:22,631 [242] [DEBUG] [app] Ending request: urn:request:917ec1ae-9230-4d79-b3e5-c25702742090 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:917ec1ae-9230-4d79-b3e5-c25702742090', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:34:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:34:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:22,636 [244] [DEBUG] [app] Starting request: urn:request:e6e728a9-c1a7-4c4e-84d5-1e8025d2be8c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:22,636 [243] [DEBUG] [app] Starting request: urn:request:0825c753-74a9-42cd-a4fc-9438dddac42a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:22,636 [244] [DEBUG] [app] Ending request: urn:request:e6e728a9-c1a7-4c4e-84d5-1e8025d2be8c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6e728a9-c1a7-4c4e-84d5-1e8025d2be8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:22,636 [243] [DEBUG] [app] Ending request: urn:request:0825c753-74a9-42cd-a4fc-9438dddac42a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0825c753-74a9-42cd-a4fc-9438dddac42a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:22,647 [239] [DEBUG] [app] Ending request: urn:request:490ad6c0-09b2-4f01-832b-7af2e304c6b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:490ad6c0-09b2-4f01-832b-7af2e304c6b2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:22,647 [241] [DEBUG] [app] Ending request: urn:request:433322b4-87f7-496b-9074-7bb959f26a5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:433322b4-87f7-496b-9074-7bb959f26a5d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:22,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:34:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:34:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:39.230038+00:00 (in 16.392178 seconds) securityworker stdout | 2023-01-17 05:34:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:52 UTC)" (scheduled at 2023-01-17 05:34:22.837365+00:00) securityworker stdout | 2023-01-17 05:34:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:34:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:34:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:48.830274+00:00 (in 24.188017 seconds) exportactionlogsworker stdout | 2023-01-17 05:34:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:24 UTC)" (scheduled at 2023-01-17 05:34:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:34:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:34:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:34:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:34:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:20.242198+00:00 (in 54.907568 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:34:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:25 UTC)" (scheduled at 2023-01-17 05:34:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:34:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:34:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:34:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:34:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:57.226424+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2023-01-17 05:34:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:57 UTC)" (scheduled at 2023-01-17 05:34:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:34:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 34, 27, 227281), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:34:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:34:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:34:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:38.735066+00:00 (in 9.999483 seconds) notificationworker stdout | 2023-01-17 05:34:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:38 UTC)" (scheduled at 2023-01-17 05:34:28.735066+00:00) notificationworker stdout | 2023-01-17 05:34:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 28, 736019), True, datetime.datetime(2023, 1, 17, 5, 34, 28, 736019), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:30,608 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:34:30,789 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:34:31,070 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:34:31,106 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:31,115 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:31,115 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:31,120 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:34:31,888 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:34:31,893 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:34:33,166 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:34:33,910 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:34:33,958 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:34:34,408 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:34:35,315 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:34:35,340 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:34:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:34:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:05.848675+00:00 (in 29.999490 seconds) gcworker stdout | 2023-01-17 05:34:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:05 UTC)" (scheduled at 2023-01-17 05:34:35.848675+00:00) gcworker stdout | 2023-01-17 05:34:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724075849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:34:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:34:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:34:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:34:35,975 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:34:35,977 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:34:35,980 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:34:36,231 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:34:36,981 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:34:37,003 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:34:37,625 [239] [DEBUG] [app] Starting request: urn:request:3388b46a-f558-4a5f-8bc1-abc15f37e78e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:37,625 [241] [DEBUG] [app] Starting request: urn:request:dc01bdce-4d07-46fb-b24e-434b6a4507e7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:37,630 [232] [DEBUG] [app] Starting request: urn:request:a1a4bede-ee13-4f70-8944-69c3847801d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,631 [232] [DEBUG] [app] Ending request: urn:request:a1a4bede-ee13-4f70-8944-69c3847801d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1a4bede-ee13-4f70-8944-69c3847801d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,631 [242] [DEBUG] [app] Starting request: urn:request:1c621bde-edd6-4020-88c4-2de1829dd81b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:34:37,631 [242] [DEBUG] [app] Ending request: urn:request:1c621bde-edd6-4020-88c4-2de1829dd81b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c621bde-edd6-4020-88c4-2de1829dd81b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:34:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:34:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:37,636 [244] [DEBUG] [app] Starting request: urn:request:2825f534-0165-4730-9303-a502e707b2f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:37,636 [243] [DEBUG] [app] Starting request: urn:request:0e6da684-f3fe-424b-8fa5-8630a2eaff51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:37,636 [244] [DEBUG] [app] Ending request: urn:request:2825f534-0165-4730-9303-a502e707b2f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2825f534-0165-4730-9303-a502e707b2f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:37,636 [243] [DEBUG] [app] Ending request: urn:request:0e6da684-f3fe-424b-8fa5-8630a2eaff51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0e6da684-f3fe-424b-8fa5-8630a2eaff51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:37,647 [239] [DEBUG] [app] Ending request: urn:request:3388b46a-f558-4a5f-8bc1-abc15f37e78e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3388b46a-f558-4a5f-8bc1-abc15f37e78e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:37,647 [241] [DEBUG] [app] Ending request: urn:request:dc01bdce-4d07-46fb-b24e-434b6a4507e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc01bdce-4d07-46fb-b24e-434b6a4507e7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:34:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:34:38,250 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:34:38,328 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:34:38,340 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:34:38,343 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:34:38,346 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:34:38,348 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:34:38,357 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:34:38,359 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:34:38,362 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:34:38,365 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:34:38,520 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:34:38,522 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:34:38,525 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:34:38,527 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:34:38,530 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:34:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:43.935629+00:00 (in 5.200094 seconds) notificationworker stdout | 2023-01-17 05:34:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:48 UTC)" (scheduled at 2023-01-17 05:34:38.735066+00:00) notificationworker stdout | 2023-01-17 05:34:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 38, 735942), True, datetime.datetime(2023, 1, 17, 5, 34, 38, 735942), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:38,761 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} repositorygcworker stdout | 2023-01-17 05:34:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:34:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:48.738458+00:00 (in 9.707230 seconds) repositorygcworker stdout | 2023-01-17 05:34:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:39 UTC)" (scheduled at 2023-01-17 05:34:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:34:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:34:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 39, 31652), True, datetime.datetime(2023, 1, 17, 5, 34, 39, 31652), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:34:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:34:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:34:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:52.837365+00:00 (in 13.606873 seconds) securityworker stdout | 2023-01-17 05:34:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:09 UTC)" (scheduled at 2023-01-17 05:34:39.230038+00:00) securityworker stdout | 2023-01-17 05:34:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:34:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:34:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:34:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:34:39,587 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:34:39,605 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:34:39,642 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:34:39,670 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:34:41,131 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:41,145 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:41,145 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:41,150 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:48.735066+00:00 (in 4.798903 seconds) notificationworker stdout | 2023-01-17 05:34:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:43 UTC)" (scheduled at 2023-01-17 05:34:43.935629+00:00) notificationworker stdout | 2023-01-17 05:34:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:34:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:34:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:34:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:59.337413+00:00 (in 11.404470 seconds) namespacegcworker stdout | 2023-01-17 05:34:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:47 UTC)" (scheduled at 2023-01-17 05:34:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:34:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:34:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:58.735066+00:00 (in 9.999391 seconds) notificationworker stdout | 2023-01-17 05:34:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:58 UTC)" (scheduled at 2023-01-17 05:34:48.735066+00:00) notificationworker stdout | 2023-01-17 05:34:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 48, 736055), True, datetime.datetime(2023, 1, 17, 5, 34, 48, 736055), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:34:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:34:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:39.030814+00:00 (in 50.291894 seconds) repositorygcworker stdout | 2023-01-17 05:34:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:48 UTC)" (scheduled at 2023-01-17 05:34:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:34:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:34:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:34:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:34:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:24.641816+00:00 (in 35.811055 seconds) exportactionlogsworker stdout | 2023-01-17 05:34:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:48 UTC)" (scheduled at 2023-01-17 05:34:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:34:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:34:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 48, 831143), True, datetime.datetime(2023, 1, 17, 5, 34, 48, 831143), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:34:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:34:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:34:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:48 UTC)" executed successfully builder stdout | 2023-01-17 05:34:51,154 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:51,163 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:51,163 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:51,169 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:52,625 [243] [DEBUG] [app] Starting request: urn:request:001d1402-bd1f-4dfb-81a4-f9a81850f18b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:52,626 [244] [DEBUG] [app] Starting request: urn:request:60820d5d-4b7a-4254-8074-484a370227bf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:34:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:52,630 [237] [DEBUG] [app] Starting request: urn:request:d27c319b-7667-474f-94ee-fbf56cfb8329 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,631 [242] [DEBUG] [app] Starting request: urn:request:2d04dc0d-d36f-4037-bb8e-1a9c6d10de83 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,631 [237] [DEBUG] [app] Ending request: urn:request:d27c319b-7667-474f-94ee-fbf56cfb8329 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d27c319b-7667-474f-94ee-fbf56cfb8329', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,631 [242] [DEBUG] [app] Ending request: urn:request:2d04dc0d-d36f-4037-bb8e-1a9c6d10de83 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d04dc0d-d36f-4037-bb8e-1a9c6d10de83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:34:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:34:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:52,635 [241] [DEBUG] [app] Starting request: urn:request:1984cfa6-b21c-42e5-8774-8e6a3bb34999 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:52,636 [239] [DEBUG] [app] Starting request: urn:request:1ca8782f-959e-4198-afcf-870a693485d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:52,636 [241] [DEBUG] [app] Ending request: urn:request:1984cfa6-b21c-42e5-8774-8e6a3bb34999 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1984cfa6-b21c-42e5-8774-8e6a3bb34999', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:34:52,636 [239] [DEBUG] [app] Ending request: urn:request:1ca8782f-959e-4198-afcf-870a693485d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1ca8782f-959e-4198-afcf-870a693485d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:52,647 [244] [DEBUG] [app] Ending request: urn:request:60820d5d-4b7a-4254-8074-484a370227bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:60820d5d-4b7a-4254-8074-484a370227bf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:52,647 [243] [DEBUG] [app] Ending request: urn:request:001d1402-bd1f-4dfb-81a4-f9a81850f18b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:001d1402-bd1f-4dfb-81a4-f9a81850f18b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:34:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:34:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:09.230038+00:00 (in 16.392191 seconds) securityworker stdout | 2023-01-17 05:34:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:22 UTC)" (scheduled at 2023-01-17 05:34:52.837365+00:00) securityworker stdout | 2023-01-17 05:34:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:34:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:34:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:27.226424+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 05:34:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:27 UTC)" (scheduled at 2023-01-17 05:34:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:34:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 34, 57, 227336), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:34:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:34:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:34:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:08.735066+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 05:34:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:08 UTC)" (scheduled at 2023-01-17 05:34:58.735066+00:00) notificationworker stdout | 2023-01-17 05:34:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 58, 735942), True, datetime.datetime(2023, 1, 17, 5, 34, 58, 735942), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:58,747 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:58,747 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:58,747 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:34:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:34:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:47.932490+00:00 (in 48.594605 seconds) namespacegcworker stdout | 2023-01-17 05:34:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:59 UTC)" (scheduled at 2023-01-17 05:34:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:34:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:34:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 59, 338335), True, datetime.datetime(2023, 1, 17, 5, 34, 59, 338335), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:34:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:34:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:34:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:00,644 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:35:00,796 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:35:01,093 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:35:01,176 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:01,185 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:01,186 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:01,191 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:35:01,905 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:35:01,924 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:35:03,202 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:35:03,930 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:35:03,994 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:35:04,436 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:35:05,331 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:35:05,348 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:35:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:35:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:35.848675+00:00 (in 29.999484 seconds) gcworker stdout | 2023-01-17 05:35:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:35 UTC)" (scheduled at 2023-01-17 05:35:05.848675+00:00) gcworker stdout | 2023-01-17 05:35:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724105849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:35:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:35:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:35:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:35:05,983 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:35:05,985 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:35:05,987 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:35:06,243 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:35:07,017 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:35:07,020 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:35:07,626 [239] [DEBUG] [app] Starting request: urn:request:23d5d824-3b3f-4a26-ae68-69122d8937f4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:07,626 [244] [DEBUG] [app] Starting request: urn:request:3d37575e-6a9c-4304-a6b7-0ae5917a1881 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:07,631 [232] [DEBUG] [app] Starting request: urn:request:cdc1e404-a887-48c0-9b33-46779a34e2d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,631 [237] [DEBUG] [app] Starting request: urn:request:5290f77b-1c07-4f07-9465-ec7befa31290 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,631 [232] [DEBUG] [app] Ending request: urn:request:cdc1e404-a887-48c0-9b33-46779a34e2d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdc1e404-a887-48c0-9b33-46779a34e2d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,631 [237] [DEBUG] [app] Ending request: urn:request:5290f77b-1c07-4f07-9465-ec7befa31290 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5290f77b-1c07-4f07-9465-ec7befa31290', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:35:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:35:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:07,637 [243] [DEBUG] [app] Starting request: urn:request:fa1cf15a-e77e-4226-8480-2a69d8055c28 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:07,637 [241] [DEBUG] [app] Starting request: urn:request:94198d09-6dcd-4796-8fcd-5093f43ae5f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:07,637 [243] [DEBUG] [app] Ending request: urn:request:fa1cf15a-e77e-4226-8480-2a69d8055c28 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa1cf15a-e77e-4226-8480-2a69d8055c28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:35:07,637 [241] [DEBUG] [app] Ending request: urn:request:94198d09-6dcd-4796-8fcd-5093f43ae5f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94198d09-6dcd-4796-8fcd-5093f43ae5f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:07,648 [239] [DEBUG] [app] Ending request: urn:request:23d5d824-3b3f-4a26-ae68-69122d8937f4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23d5d824-3b3f-4a26-ae68-69122d8937f4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:07,648 [244] [DEBUG] [app] Ending request: urn:request:3d37575e-6a9c-4304-a6b7-0ae5917a1881 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d37575e-6a9c-4304-a6b7-0ae5917a1881', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:07,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:07,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:07,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:35:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) storagereplication stdout | 2023-01-17 05:35:08,280 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:35:08,337 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:35:08,345 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:35:08,352 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:35:08,355 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:35:08,357 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:35:08,364 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:35:08,367 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:35:08,370 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:35:08,372 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:35:08,527 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:35:08,529 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:35:08,532 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:35:08,534 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:35:08,537 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:35:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:18.735066+00:00 (in 9.999565 seconds) notificationworker stdout | 2023-01-17 05:35:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:18 UTC)" (scheduled at 2023-01-17 05:35:08.735066+00:00) notificationworker stdout | 2023-01-17 05:35:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 8, 735915), True, datetime.datetime(2023, 1, 17, 5, 35, 8, 735915), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:08,785 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:35:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:22.837365+00:00 (in 13.606862 seconds) securityworker stdout | 2023-01-17 05:35:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:39 UTC)" (scheduled at 2023-01-17 05:35:09.230038+00:00) securityworker stdout | 2023-01-17 05:35:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:35:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:35:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:35:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:35:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:35:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:35:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:35:09,616 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:35:09,639 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:35:09,674 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:35:09,700 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:35:11,201 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:11,211 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:11,211 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:11,216 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:28.735066+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 05:35:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:28 UTC)" (scheduled at 2023-01-17 05:35:18.735066+00:00) notificationworker stdout | 2023-01-17 05:35:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 18, 735916), True, datetime.datetime(2023, 1, 17, 5, 35, 18, 735916), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:35:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:35:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:25.334138+00:00 (in 5.091468 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:35:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:20 UTC)" (scheduled at 2023-01-17 05:35:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:35:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:35:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 20, 243058), True, datetime.datetime(2023, 1, 17, 5, 35, 20, 243058), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:35:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:35:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:35:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:20 UTC)" executed successfully builder stdout | 2023-01-17 05:35:21,225 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:21,234 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:21,234 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:21,240 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:22,625 [239] [DEBUG] [app] Starting request: urn:request:ba46bf5b-05d1-4a35-8240-bbd7dc7c3a38 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:22,625 [244] [DEBUG] [app] Starting request: urn:request:ca2d44b0-419d-4fdf-bd52-ecd5cae86fcf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:22,630 [240] [DEBUG] [app] Starting request: urn:request:2fe5a729-40fa-438d-b7e8-a356b8b718d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,631 [242] [DEBUG] [app] Starting request: urn:request:4ce5dd64-06dc-4140-a708-de0a4a2a1c63 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,631 [240] [DEBUG] [app] Ending request: urn:request:2fe5a729-40fa-438d-b7e8-a356b8b718d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2fe5a729-40fa-438d-b7e8-a356b8b718d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:35:22,631 [242] [DEBUG] [app] Ending request: urn:request:4ce5dd64-06dc-4140-a708-de0a4a2a1c63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ce5dd64-06dc-4140-a708-de0a4a2a1c63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:35:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:35:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:22,635 [241] [DEBUG] [app] Starting request: urn:request:89fa60bb-d03b-405c-b21b-cc96c9c6319e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:22,635 [244] [DEBUG] [app] Starting request: urn:request:a619459c-6fb2-400b-b537-dbd8a8e332a9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:22,636 [244] [DEBUG] [app] Ending request: urn:request:a619459c-6fb2-400b-b537-dbd8a8e332a9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a619459c-6fb2-400b-b537-dbd8a8e332a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:22,636 [241] [DEBUG] [app] Ending request: urn:request:89fa60bb-d03b-405c-b21b-cc96c9c6319e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89fa60bb-d03b-405c-b21b-cc96c9c6319e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:35:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:22,647 [244] [DEBUG] [app] Ending request: urn:request:ca2d44b0-419d-4fdf-bd52-ecd5cae86fcf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca2d44b0-419d-4fdf-bd52-ecd5cae86fcf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:22,647 [239] [DEBUG] [app] Ending request: urn:request:ba46bf5b-05d1-4a35-8240-bbd7dc7c3a38 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba46bf5b-05d1-4a35-8240-bbd7dc7c3a38', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:35:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:35:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:39.230038+00:00 (in 16.392235 seconds) securityworker stdout | 2023-01-17 05:35:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:52 UTC)" (scheduled at 2023-01-17 05:35:22.837365+00:00) securityworker stdout | 2023-01-17 05:35:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:35:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:35:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:48.830274+00:00 (in 24.188007 seconds) exportactionlogsworker stdout | 2023-01-17 05:35:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:24 UTC)" (scheduled at 2023-01-17 05:35:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:35:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:35:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:35:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:35:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:20.242198+00:00 (in 54.907573 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:35:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:25 UTC)" (scheduled at 2023-01-17 05:35:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:35:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:35:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:35:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:35:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:57.226424+00:00 (in 29.999476 seconds) buildlogsarchiver stdout | 2023-01-17 05:35:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:57 UTC)" (scheduled at 2023-01-17 05:35:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:35:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 35, 27, 227353), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:35:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:35:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:35:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:38.735066+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:35:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:38 UTC)" (scheduled at 2023-01-17 05:35:28.735066+00:00) notificationworker stdout | 2023-01-17 05:35:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 28, 735902), True, datetime.datetime(2023, 1, 17, 5, 35, 28, 735902), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:30,678 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:35:30,833 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:35:31,128 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:35:31,248 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:31,257 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:31,257 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:31,263 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:35:31,934 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:35:31,958 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:35:33,237 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:35:33,945 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:35:34,029 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:35:34,446 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:35:35,343 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:35:35,382 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:35:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:35:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:05.848675+00:00 (in 29.999489 seconds) gcworker stdout | 2023-01-17 05:35:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:05 UTC)" (scheduled at 2023-01-17 05:35:35.848675+00:00) gcworker stdout | 2023-01-17 05:35:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724135849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:35:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:35:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:35:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:35:35,990 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:35:35,992 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:35:35,995 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:35:36,268 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:35:37,036 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:35:37,052 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:35:37,625 [244] [DEBUG] [app] Starting request: urn:request:1de1b1e2-824b-4dad-ac6c-610206a687f2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:37,625 [243] [DEBUG] [app] Starting request: urn:request:62cdd1b6-c7b5-488c-a4f3-dcfa11a48768 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:37,630 [242] [DEBUG] [app] Starting request: urn:request:20b7b490-e9d6-4f5c-8171-e2916916c170 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:37,630 [232] [DEBUG] [app] Starting request: urn:request:8585df41-0bda-4767-b48c-c02c731917d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:37,630 [242] [DEBUG] [app] Ending request: urn:request:20b7b490-e9d6-4f5c-8171-e2916916c170 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20b7b490-e9d6-4f5c-8171-e2916916c170', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:35:37,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:35:37,630 [232] [DEBUG] [app] Ending request: urn:request:8585df41-0bda-4767-b48c-c02c731917d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8585df41-0bda-4767-b48c-c02c731917d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:37,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:35:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:35:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:37,635 [239] [DEBUG] [app] Starting request: urn:request:800a9c98-ed17-4599-957e-e3eb7294a843 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:37,635 [241] [DEBUG] [app] Starting request: urn:request:588772fd-fb0c-49d4-851a-ca9640ecaff1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:37,635 [239] [DEBUG] [app] Ending request: urn:request:800a9c98-ed17-4599-957e-e3eb7294a843 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:800a9c98-ed17-4599-957e-e3eb7294a843', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:37,636 [241] [DEBUG] [app] Ending request: urn:request:588772fd-fb0c-49d4-851a-ca9640ecaff1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:588772fd-fb0c-49d4-851a-ca9640ecaff1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:35:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:37,646 [243] [DEBUG] [app] Ending request: urn:request:62cdd1b6-c7b5-488c-a4f3-dcfa11a48768 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62cdd1b6-c7b5-488c-a4f3-dcfa11a48768', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:37,647 [244] [DEBUG] [app] Ending request: urn:request:1de1b1e2-824b-4dad-ac6c-610206a687f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1de1b1e2-824b-4dad-ac6c-610206a687f2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:35:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) storagereplication stdout | 2023-01-17 05:35:38,316 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:35:38,344 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:35:38,354 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:35:38,358 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:35:38,361 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:35:38,364 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:35:38,373 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:35:38,375 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:35:38,378 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:35:38,381 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:35:38,533 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:35:38,536 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:35:38,539 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:35:38,542 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:35:38,544 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:35:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:43.935629+00:00 (in 5.200081 seconds) notificationworker stdout | 2023-01-17 05:35:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:48 UTC)" (scheduled at 2023-01-17 05:35:38.735066+00:00) notificationworker stdout | 2023-01-17 05:35:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 38, 735936), True, datetime.datetime(2023, 1, 17, 5, 35, 38, 735936), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:38,818 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} repositorygcworker stdout | 2023-01-17 05:35:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:35:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:48.738458+00:00 (in 9.707126 seconds) repositorygcworker stdout | 2023-01-17 05:35:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:39 UTC)" (scheduled at 2023-01-17 05:35:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:35:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:35:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 39, 31749), True, datetime.datetime(2023, 1, 17, 5, 35, 39, 31749), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:35:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:35:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:35:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:52.837365+00:00 (in 13.606904 seconds) securityworker stdout | 2023-01-17 05:35:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:09 UTC)" (scheduled at 2023-01-17 05:35:39.230038+00:00) securityworker stdout | 2023-01-17 05:35:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:35:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:35:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:35:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:35:39,646 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:35:39,674 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:35:39,697 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:35:39,707 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:35:41,264 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:41,293 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:41,293 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:41,299 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:48.735066+00:00 (in 4.798940 seconds) notificationworker stdout | 2023-01-17 05:35:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:43 UTC)" (scheduled at 2023-01-17 05:35:43.935629+00:00) notificationworker stdout | 2023-01-17 05:35:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:35:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:35:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:35:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:59.337413+00:00 (in 11.404436 seconds) namespacegcworker stdout | 2023-01-17 05:35:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:47 UTC)" (scheduled at 2023-01-17 05:35:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:35:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:35:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:58.735066+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:35:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:58 UTC)" (scheduled at 2023-01-17 05:35:48.735066+00:00) notificationworker stdout | 2023-01-17 05:35:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 48, 735914), True, datetime.datetime(2023, 1, 17, 5, 35, 48, 735914), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:35:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:35:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:39.030814+00:00 (in 50.291913 seconds) repositorygcworker stdout | 2023-01-17 05:35:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:48 UTC)" (scheduled at 2023-01-17 05:35:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:35:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:35:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:35:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:35:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:24.641816+00:00 (in 35.811062 seconds) exportactionlogsworker stdout | 2023-01-17 05:35:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:48 UTC)" (scheduled at 2023-01-17 05:35:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:35:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:35:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 48, 831136), True, datetime.datetime(2023, 1, 17, 5, 35, 48, 831136), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:35:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:35:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:35:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:48 UTC)" executed successfully builder stdout | 2023-01-17 05:35:51,309 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:51,318 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:51,318 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:51,324 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:52,625 [239] [DEBUG] [app] Starting request: urn:request:886116b7-efaa-4f11-8720-ed4b9d558ccc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:52,626 [244] [DEBUG] [app] Starting request: urn:request:cde857af-6ced-475a-a57e-904c152763e7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:35:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:52,630 [232] [DEBUG] [app] Starting request: urn:request:6cda43fd-0ba6-4a8c-8537-200b7c8e46a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:52,630 [242] [DEBUG] [app] Starting request: urn:request:6b798d91-18e0-4af7-af7c-d5bd169dd64c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:52,630 [232] [DEBUG] [app] Ending request: urn:request:6cda43fd-0ba6-4a8c-8537-200b7c8e46a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6cda43fd-0ba6-4a8c-8537-200b7c8e46a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:35:52,631 [242] [DEBUG] [app] Ending request: urn:request:6b798d91-18e0-4af7-af7c-d5bd169dd64c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b798d91-18e0-4af7-af7c-d5bd169dd64c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:35:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:35:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:52,635 [243] [DEBUG] [app] Starting request: urn:request:feef722e-bd0a-40d0-bc6c-ee5ef2680555 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:52,635 [239] [DEBUG] [app] Starting request: urn:request:35ac83d2-33ae-48ea-9bde-6530cdbb8085 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:52,636 [243] [DEBUG] [app] Ending request: urn:request:feef722e-bd0a-40d0-bc6c-ee5ef2680555 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:feef722e-bd0a-40d0-bc6c-ee5ef2680555', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:52,636 [239] [DEBUG] [app] Ending request: urn:request:35ac83d2-33ae-48ea-9bde-6530cdbb8085 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:35ac83d2-33ae-48ea-9bde-6530cdbb8085', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:52,652 [239] [DEBUG] [app] Ending request: urn:request:886116b7-efaa-4f11-8720-ed4b9d558ccc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:886116b7-efaa-4f11-8720-ed4b9d558ccc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:52,652 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:52,652 [244] [DEBUG] [app] Ending request: urn:request:cde857af-6ced-475a-a57e-904c152763e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cde857af-6ced-475a-a57e-904c152763e7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:52,652 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:52,653 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:35:52,653 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) securityworker stdout | 2023-01-17 05:35:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:09.230038+00:00 (in 16.392231 seconds) securityworker stdout | 2023-01-17 05:35:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:22 UTC)" (scheduled at 2023-01-17 05:35:52.837365+00:00) securityworker stdout | 2023-01-17 05:35:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:35:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:35:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:27.226424+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2023-01-17 05:35:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:27 UTC)" (scheduled at 2023-01-17 05:35:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:35:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 35, 57, 227279), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:35:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:35:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:35:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:08.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 05:35:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:08 UTC)" (scheduled at 2023-01-17 05:35:58.735066+00:00) notificationworker stdout | 2023-01-17 05:35:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 58, 735955), True, datetime.datetime(2023, 1, 17, 5, 35, 58, 735955), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:35:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:35:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:47.932490+00:00 (in 48.594641 seconds) namespacegcworker stdout | 2023-01-17 05:35:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:59 UTC)" (scheduled at 2023-01-17 05:35:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:35:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:35:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 59, 338266), True, datetime.datetime(2023, 1, 17, 5, 35, 59, 338266), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:35:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:35:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:35:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:00,713 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:36:00,864 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:36:01,159 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:36:01,334 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:01,343 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:01,343 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:01,349 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:36:01,969 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:36:01,993 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:36:03,273 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:36:03,981 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:36:04,064 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:36:04,481 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:36:05,371 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:36:05,417 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:36:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:36:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:35.848675+00:00 (in 29.999497 seconds) gcworker stdout | 2023-01-17 05:36:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:35 UTC)" (scheduled at 2023-01-17 05:36:05.848675+00:00) gcworker stdout | 2023-01-17 05:36:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724165849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:36:05,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:36:05,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:36:05,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:36:05,997 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:36:06,000 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:36:06,002 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:36:06,294 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:36:07,042 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:36:07,059 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:36:07,626 [241] [DEBUG] [app] Starting request: urn:request:572fd77f-731c-4481-934f-e3337f2a4619 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:07,626 [239] [DEBUG] [app] Starting request: urn:request:018f53be-f0d9-4c9b-bdbd-823e37d6896b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:07,631 [240] [DEBUG] [app] Starting request: urn:request:4ca482f9-feca-4e00-b3b7-2857df34afd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,631 [232] [DEBUG] [app] Starting request: urn:request:29d1ccd4-7257-4f4f-a80c-88dfeb1a7a4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,631 [240] [DEBUG] [app] Ending request: urn:request:4ca482f9-feca-4e00-b3b7-2857df34afd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ca482f9-feca-4e00-b3b7-2857df34afd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,631 [232] [DEBUG] [app] Ending request: urn:request:29d1ccd4-7257-4f4f-a80c-88dfeb1a7a4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29d1ccd4-7257-4f4f-a80c-88dfeb1a7a4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:36:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:36:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:07,636 [244] [DEBUG] [app] Starting request: urn:request:00d367f7-a955-4605-9659-1d9b4982446e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:07,636 [239] [DEBUG] [app] Starting request: urn:request:dbd429db-be7b-4853-a120-57acf76e08a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:07,636 [239] [DEBUG] [app] Ending request: urn:request:dbd429db-be7b-4853-a120-57acf76e08a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dbd429db-be7b-4853-a120-57acf76e08a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:07,636 [244] [DEBUG] [app] Ending request: urn:request:00d367f7-a955-4605-9659-1d9b4982446e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00d367f7-a955-4605-9659-1d9b4982446e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:07,647 [239] [DEBUG] [app] Ending request: urn:request:018f53be-f0d9-4c9b-bdbd-823e37d6896b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:018f53be-f0d9-4c9b-bdbd-823e37d6896b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:07,647 [241] [DEBUG] [app] Ending request: urn:request:572fd77f-731c-4481-934f-e3337f2a4619 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:572fd77f-731c-4481-934f-e3337f2a4619', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:07,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:36:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 05:36:08,351 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:36:08,353 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:36:08,364 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:36:08,366 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:36:08,369 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:36:08,371 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:36:08,379 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:36:08,382 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:36:08,388 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:36:08,390 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:36:08,541 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:36:08,543 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:36:08,545 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:36:08,548 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:36:08,550 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:36:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:18.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:36:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:18 UTC)" (scheduled at 2023-01-17 05:36:08.735066+00:00) notificationworker stdout | 2023-01-17 05:36:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 8, 735909), True, datetime.datetime(2023, 1, 17, 5, 36, 8, 735909), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:08,853 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:36:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:22.837365+00:00 (in 13.606871 seconds) securityworker stdout | 2023-01-17 05:36:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:39 UTC)" (scheduled at 2023-01-17 05:36:09.230038+00:00) securityworker stdout | 2023-01-17 05:36:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:36:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:36:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:36:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:36:09,678 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:36:09,708 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:36:09,719 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:36:09,741 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:36:11,359 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:11,368 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:11,368 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:11,374 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:28.735066+00:00 (in 9.999571 seconds) notificationworker stdout | 2023-01-17 05:36:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:28 UTC)" (scheduled at 2023-01-17 05:36:18.735066+00:00) notificationworker stdout | 2023-01-17 05:36:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 18, 735920), True, datetime.datetime(2023, 1, 17, 5, 36, 18, 735920), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:36:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:36:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:25.334138+00:00 (in 5.091488 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:36:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:20 UTC)" (scheduled at 2023-01-17 05:36:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:36:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:36:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 20, 243044), True, datetime.datetime(2023, 1, 17, 5, 36, 20, 243044), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:36:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:36:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:36:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:20 UTC)" executed successfully builder stdout | 2023-01-17 05:36:21,375 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:21,383 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:21,383 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:21,389 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:22,626 [239] [DEBUG] [app] Starting request: urn:request:80dbdc1a-66ef-4677-89f5-897b0e8d44ca (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:22,626 [243] [DEBUG] [app] Starting request: urn:request:2fca7e5d-c50e-4ca4-9ef0-56797c84f0ad (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:22,631 [242] [DEBUG] [app] Starting request: urn:request:42d2f608-489a-47f8-b475-99b13edf0fbc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,631 [242] [DEBUG] [app] Ending request: urn:request:42d2f608-489a-47f8-b475-99b13edf0fbc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:42d2f608-489a-47f8-b475-99b13edf0fbc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,631 [232] [DEBUG] [app] Starting request: urn:request:6c4c09ac-d2bd-4403-a922-abd35e54b428 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:36:22,631 [232] [DEBUG] [app] Ending request: urn:request:6c4c09ac-d2bd-4403-a922-abd35e54b428 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c4c09ac-d2bd-4403-a922-abd35e54b428', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:36:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:22,636 [241] [DEBUG] [app] Starting request: urn:request:a784292a-3535-4dfb-8201-44b938407ed5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:22,636 [244] [DEBUG] [app] Starting request: urn:request:64889dcf-1bbc-4dad-a1be-2c6fa07c9695 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:22,636 [241] [DEBUG] [app] Ending request: urn:request:a784292a-3535-4dfb-8201-44b938407ed5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a784292a-3535-4dfb-8201-44b938407ed5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:22,636 [244] [DEBUG] [app] Ending request: urn:request:64889dcf-1bbc-4dad-a1be-2c6fa07c9695 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64889dcf-1bbc-4dad-a1be-2c6fa07c9695', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:22,647 [239] [DEBUG] [app] Ending request: urn:request:80dbdc1a-66ef-4677-89f5-897b0e8d44ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80dbdc1a-66ef-4677-89f5-897b0e8d44ca', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:22,647 [243] [DEBUG] [app] Ending request: urn:request:2fca7e5d-c50e-4ca4-9ef0-56797c84f0ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2fca7e5d-c50e-4ca4-9ef0-56797c84f0ad', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:36:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:36:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:39.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 05:36:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:52 UTC)" (scheduled at 2023-01-17 05:36:22.837365+00:00) securityworker stdout | 2023-01-17 05:36:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:36:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:36:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:44.641497+00:00 (in 19.999206 seconds) exportactionlogsworker stdout | 2023-01-17 05:36:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:24 UTC)" (scheduled at 2023-01-17 05:36:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:36:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:36:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:36:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:36:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:39.333860+00:00 (in 13.999260 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:36:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:25 UTC)" (scheduled at 2023-01-17 05:36:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:36:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:36:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:36:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:36:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:57.226424+00:00 (in 29.999557 seconds) buildlogsarchiver stdout | 2023-01-17 05:36:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:57 UTC)" (scheduled at 2023-01-17 05:36:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:36:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 36, 27, 227195), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:36:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:36:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:36:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:38.735066+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:36:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:38 UTC)" (scheduled at 2023-01-17 05:36:28.735066+00:00) notificationworker stdout | 2023-01-17 05:36:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 28, 735978), True, datetime.datetime(2023, 1, 17, 5, 36, 28, 735978), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:30,748 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:36:30,893 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:36:31,164 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:36:31,399 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:31,407 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:31,408 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:31,413 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:36:32,005 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:36:32,027 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:36:33,309 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:36:33,997 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:36:34,099 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:36:34,493 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:36:35,408 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:36:35,433 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:36:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:36:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:05.848675+00:00 (in 29.999501 seconds) gcworker stdout | 2023-01-17 05:36:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:05 UTC)" (scheduled at 2023-01-17 05:36:35.848675+00:00) gcworker stdout | 2023-01-17 05:36:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724195849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:36:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:36:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:36:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:36:36,004 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:36:36,007 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:36:36,009 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:36:36,301 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:36:37,078 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:36:37,094 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:36:37,625 [243] [DEBUG] [app] Starting request: urn:request:d5ff7b2c-c940-420d-8c99-8ac6d46ba923 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:37,625 [239] [DEBUG] [app] Starting request: urn:request:acd3d0f1-5976-40cb-bf4f-535daf1ddcec (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:37,630 [242] [DEBUG] [app] Starting request: urn:request:9a6d163c-dec3-485a-ba7f-56b5c5b7244a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:37,631 [237] [DEBUG] [app] Starting request: urn:request:e281121a-5d21-44ab-b91b-eb01bbe6b850 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:37,631 [242] [DEBUG] [app] Ending request: urn:request:9a6d163c-dec3-485a-ba7f-56b5c5b7244a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a6d163c-dec3-485a-ba7f-56b5c5b7244a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:36:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:36:37,631 [237] [DEBUG] [app] Ending request: urn:request:e281121a-5d21-44ab-b91b-eb01bbe6b850 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e281121a-5d21-44ab-b91b-eb01bbe6b850', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:36:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:36:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:37,635 [244] [DEBUG] [app] Starting request: urn:request:640e7f2b-4f1f-4a53-a50e-3e9f549087d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:37,635 [244] [DEBUG] [app] Ending request: urn:request:640e7f2b-4f1f-4a53-a50e-3e9f549087d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:640e7f2b-4f1f-4a53-a50e-3e9f549087d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:37,636 [241] [DEBUG] [app] Starting request: urn:request:c4c839f5-1704-45b5-9b3a-2eecd0bc9bf9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:37,636 [241] [DEBUG] [app] Ending request: urn:request:c4c839f5-1704-45b5-9b3a-2eecd0bc9bf9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4c839f5-1704-45b5-9b3a-2eecd0bc9bf9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:37,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:36:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:37,646 [243] [DEBUG] [app] Ending request: urn:request:d5ff7b2c-c940-420d-8c99-8ac6d46ba923 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5ff7b2c-c940-420d-8c99-8ac6d46ba923', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:37,647 [239] [DEBUG] [app] Ending request: urn:request:acd3d0f1-5976-40cb-bf4f-535daf1ddcec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:acd3d0f1-5976-40cb-bf4f-535daf1ddcec', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:36:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:36:38,361 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:36:38,372 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:36:38,374 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:36:38,378 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:36:38,380 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} storagereplication stdout | 2023-01-17 05:36:38,386 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:36:38,388 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:36:38,391 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:36:38,394 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:36:38,399 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:36:38,549 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:36:38,552 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:36:38,554 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:36:38,557 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:36:38,560 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:36:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:43.935629+00:00 (in 5.200112 seconds) notificationworker stdout | 2023-01-17 05:36:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:48 UTC)" (scheduled at 2023-01-17 05:36:38.735066+00:00) notificationworker stdout | 2023-01-17 05:36:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 38, 735908), True, datetime.datetime(2023, 1, 17, 5, 36, 38, 735908), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:38,869 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} repositorygcworker stdout | 2023-01-17 05:36:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:36:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:48.738458+00:00 (in 9.707178 seconds) repositorygcworker stdout | 2023-01-17 05:36:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:39 UTC)" (scheduled at 2023-01-17 05:36:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:36:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:36:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 39, 31697), True, datetime.datetime(2023, 1, 17, 5, 36, 39, 31697), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:36:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:36:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:36:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:52.837365+00:00 (in 13.606879 seconds) securityworker stdout | 2023-01-17 05:36:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:09 UTC)" (scheduled at 2023-01-17 05:36:39.230038+00:00) securityworker stdout | 2023-01-17 05:36:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:36:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:36:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:36:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:36:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:36:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:20.242198+00:00 (in 40.907872 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:36:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:39 UTC)" (scheduled at 2023-01-17 05:36:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:36:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 36, 39, 334676), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:36:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 39, 334676), True, datetime.datetime(2023, 1, 17, 5, 36, 39, 334676), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:36:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 39, 334676), True, datetime.datetime(2023, 1, 17, 5, 36, 39, 334676), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 36, 39, 334676), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:36:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:36:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:36:39,701 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:36:39,731 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:36:39,735 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:36:39,776 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:36:41,423 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:41,432 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:41,432 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:41,438 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:48.735066+00:00 (in 4.798964 seconds) notificationworker stdout | 2023-01-17 05:36:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:43 UTC)" (scheduled at 2023-01-17 05:36:43.935629+00:00) notificationworker stdout | 2023-01-17 05:36:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:36:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:36:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:36:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:48.830274+00:00 (in 4.188276 seconds) exportactionlogsworker stdout | 2023-01-17 05:36:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:44 UTC)" (scheduled at 2023-01-17 05:36:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:36:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 36, 44, 642391), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:36:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 44, 642391), True, datetime.datetime(2023, 1, 17, 5, 36, 44, 642391), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:36:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 44, 642391), True, datetime.datetime(2023, 1, 17, 5, 36, 44, 642391), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 36, 44, 642391), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:36:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:36:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:36:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:36:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:59.337413+00:00 (in 11.404446 seconds) namespacegcworker stdout | 2023-01-17 05:36:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:47 UTC)" (scheduled at 2023-01-17 05:36:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:36:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:36:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:58.735066+00:00 (in 9.999485 seconds) notificationworker stdout | 2023-01-17 05:36:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:58 UTC)" (scheduled at 2023-01-17 05:36:48.735066+00:00) notificationworker stdout | 2023-01-17 05:36:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 48, 735970), True, datetime.datetime(2023, 1, 17, 5, 36, 48, 735970), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:36:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:36:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:04.738062+00:00 (in 15.999150 seconds) repositorygcworker stdout | 2023-01-17 05:36:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:48 UTC)" (scheduled at 2023-01-17 05:36:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:36:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:36:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:36:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:36:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:24.641816+00:00 (in 35.811109 seconds) exportactionlogsworker stdout | 2023-01-17 05:36:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:48 UTC)" (scheduled at 2023-01-17 05:36:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:36:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:36:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 48, 831101), True, datetime.datetime(2023, 1, 17, 5, 36, 48, 831101), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:36:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:36:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:36:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:48 UTC)" executed successfully builder stdout | 2023-01-17 05:36:51,448 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:51,457 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:51,457 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:51,463 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:52,625 [244] [DEBUG] [app] Starting request: urn:request:6f9cea0b-b245-4027-92be-01e9ab15682f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:52,625 [241] [DEBUG] [app] Starting request: urn:request:33e074e6-4519-4135-b980-5cbcee5b749f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:36:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:52,630 [232] [DEBUG] [app] Starting request: urn:request:01729016-bb9a-4b8c-8aac-f79746307dad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,630 [237] [DEBUG] [app] Starting request: urn:request:a54dab33-bd48-4284-b15c-f62183352fa9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,630 [232] [DEBUG] [app] Ending request: urn:request:01729016-bb9a-4b8c-8aac-f79746307dad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01729016-bb9a-4b8c-8aac-f79746307dad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,631 [237] [DEBUG] [app] Ending request: urn:request:a54dab33-bd48-4284-b15c-f62183352fa9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a54dab33-bd48-4284-b15c-f62183352fa9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:36:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:36:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:52,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:52,635 [239] [DEBUG] [app] Starting request: urn:request:632f9cfa-63a0-46d2-afd4-8228b7a67a81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:52,635 [243] [DEBUG] [app] Starting request: urn:request:c253d47f-2efb-4b93-ab16-9594dc40cc77 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:52,635 [239] [DEBUG] [app] Ending request: urn:request:632f9cfa-63a0-46d2-afd4-8228b7a67a81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:632f9cfa-63a0-46d2-afd4-8228b7a67a81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:52,636 [243] [DEBUG] [app] Ending request: urn:request:c253d47f-2efb-4b93-ab16-9594dc40cc77 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c253d47f-2efb-4b93-ab16-9594dc40cc77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:36:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:52,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:52,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:52,646 [241] [DEBUG] [app] Ending request: urn:request:33e074e6-4519-4135-b980-5cbcee5b749f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33e074e6-4519-4135-b980-5cbcee5b749f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:52,647 [244] [DEBUG] [app] Ending request: urn:request:6f9cea0b-b245-4027-92be-01e9ab15682f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f9cea0b-b245-4027-92be-01e9ab15682f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:52,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:52,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:36:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:36:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:09.230038+00:00 (in 16.392229 seconds) securityworker stdout | 2023-01-17 05:36:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:22 UTC)" (scheduled at 2023-01-17 05:36:52.837365+00:00) securityworker stdout | 2023-01-17 05:36:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:36:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:36:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:27.226424+00:00 (in 29.999535 seconds) buildlogsarchiver stdout | 2023-01-17 05:36:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:27 UTC)" (scheduled at 2023-01-17 05:36:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:36:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 36, 57, 227275), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:36:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:36:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:36:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:08.735066+00:00 (in 9.999486 seconds) notificationworker stdout | 2023-01-17 05:36:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:08 UTC)" (scheduled at 2023-01-17 05:36:58.735066+00:00) notificationworker stdout | 2023-01-17 05:36:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 58, 736036), True, datetime.datetime(2023, 1, 17, 5, 36, 58, 736036), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:36:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:36:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:47.932490+00:00 (in 48.594634 seconds) namespacegcworker stdout | 2023-01-17 05:36:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:59 UTC)" (scheduled at 2023-01-17 05:36:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:36:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:36:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 59, 338356), True, datetime.datetime(2023, 1, 17, 5, 36, 59, 338356), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:36:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:36:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:36:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:00,770 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:37:00,929 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:37:01,171 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:37:01,473 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:01,482 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:01,482 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:01,488 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:37:02,040 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:37:02,061 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:37:03,323 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:37:04,030 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:37:04,133 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:37:04,523 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositorygcworker stdout | 2023-01-17 05:37:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:37:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:39.030814+00:00 (in 34.292306 seconds) repositorygcworker stdout | 2023-01-17 05:37:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:42:04 UTC)" (scheduled at 2023-01-17 05:37:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:37:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 37, 4, 738896), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:37:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 37, 4, 738896), True, datetime.datetime(2023, 1, 17, 5, 37, 4, 738896), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:37:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 37, 4, 738896), True, datetime.datetime(2023, 1, 17, 5, 37, 4, 738896), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 37, 4, 738896), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:37:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:37:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:42:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:37:05,417 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:37:05,443 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:37:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:37:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:35.848675+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 05:37:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:35 UTC)" (scheduled at 2023-01-17 05:37:05.848675+00:00) gcworker stdout | 2023-01-17 05:37:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724225849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:37:05,868 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:37:05,868 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:37:05,868 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:37:06,013 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:37:06,016 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:37:06,018 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:37:06,325 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:37:07,088 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:37:07,130 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:37:07,625 [239] [DEBUG] [app] Starting request: urn:request:c24db02e-6a62-45dc-92cd-8bc444d9e680 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:07,626 [241] [DEBUG] [app] Starting request: urn:request:55bbc0aa-eb2c-4ca4-9f76-a1af6c6c0cdd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:07,630 [240] [DEBUG] [app] Starting request: urn:request:6bf5f852-e3e4-450a-9bde-f2bdb6d3dfcc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:07,631 [232] [DEBUG] [app] Starting request: urn:request:8779b586-8095-4aaa-9594-44d54a9ca4d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:07,631 [240] [DEBUG] [app] Ending request: urn:request:6bf5f852-e3e4-450a-9bde-f2bdb6d3dfcc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6bf5f852-e3e4-450a-9bde-f2bdb6d3dfcc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:37:07,631 [232] [DEBUG] [app] Ending request: urn:request:8779b586-8095-4aaa-9594-44d54a9ca4d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8779b586-8095-4aaa-9594-44d54a9ca4d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:37:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:37:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:07,636 [244] [DEBUG] [app] Starting request: urn:request:d3f7e6ff-00f4-49e8-aecf-e0cb6d6c145d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:07,636 [243] [DEBUG] [app] Starting request: urn:request:a0dc5d92-337a-4d44-b081-5114658c83d6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:07,636 [244] [DEBUG] [app] Ending request: urn:request:d3f7e6ff-00f4-49e8-aecf-e0cb6d6c145d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3f7e6ff-00f4-49e8-aecf-e0cb6d6c145d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:07,636 [243] [DEBUG] [app] Ending request: urn:request:a0dc5d92-337a-4d44-b081-5114658c83d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0dc5d92-337a-4d44-b081-5114658c83d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:07,647 [239] [DEBUG] [app] Ending request: urn:request:c24db02e-6a62-45dc-92cd-8bc444d9e680 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c24db02e-6a62-45dc-92cd-8bc444d9e680', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:07,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:37:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:37:07,648 [241] [DEBUG] [app] Ending request: urn:request:55bbc0aa-eb2c-4ca4-9f76-a1af6c6c0cdd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55bbc0aa-eb2c-4ca4-9f76-a1af6c6c0cdd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:07,649 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:37:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:37:08,368 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:37:08,380 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:37:08,383 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:37:08,387 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:37:08,389 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} storagereplication stdout | 2023-01-17 05:37:08,394 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:37:08,396 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:37:08,398 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:37:08,402 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:37:08,407 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:37:08,558 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:37:08,560 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:37:08,563 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:37:08,565 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:37:08,568 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:37:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:18.735066+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:37:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:18 UTC)" (scheduled at 2023-01-17 05:37:08.735066+00:00) notificationworker stdout | 2023-01-17 05:37:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 8, 735940), True, datetime.datetime(2023, 1, 17, 5, 37, 8, 735940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:08,898 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:37:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:22.837365+00:00 (in 13.606883 seconds) securityworker stdout | 2023-01-17 05:37:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:39 UTC)" (scheduled at 2023-01-17 05:37:09.230038+00:00) securityworker stdout | 2023-01-17 05:37:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:37:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:37:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:37:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:37:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:37:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:37:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:37:09,737 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:37:09,766 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:37:09,769 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:37:09,807 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:37:11,498 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:11,507 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:11,507 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:11,513 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:28.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:37:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:28 UTC)" (scheduled at 2023-01-17 05:37:18.735066+00:00) notificationworker stdout | 2023-01-17 05:37:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 18, 735951), True, datetime.datetime(2023, 1, 17, 5, 37, 18, 735951), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:37:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:37:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:25.334138+00:00 (in 5.091504 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:37:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:20 UTC)" (scheduled at 2023-01-17 05:37:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:37:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:37:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 20, 243046), True, datetime.datetime(2023, 1, 17, 5, 37, 20, 243046), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:37:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:37:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:37:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:20 UTC)" executed successfully builder stdout | 2023-01-17 05:37:21,523 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:21,532 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:21,532 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:21,537 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:22,626 [243] [DEBUG] [app] Starting request: urn:request:b026b1d0-8963-4532-870f-d513e772cdf5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:22,626 [244] [DEBUG] [app] Starting request: urn:request:8f06f73b-7f83-40e8-972f-e5501634a085 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:22,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:22,631 [240] [DEBUG] [app] Starting request: urn:request:a831ff30-3d92-4a0a-a76a-9de1d54098ca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,631 [232] [DEBUG] [app] Starting request: urn:request:cd0aab04-3cfc-4108-ad44-8bde6e9c9bd6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,632 [240] [DEBUG] [app] Ending request: urn:request:a831ff30-3d92-4a0a-a76a-9de1d54098ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a831ff30-3d92-4a0a-a76a-9de1d54098ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,632 [232] [DEBUG] [app] Ending request: urn:request:cd0aab04-3cfc-4108-ad44-8bde6e9c9bd6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd0aab04-3cfc-4108-ad44-8bde6e9c9bd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:37:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:37:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:22,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:22,637 [239] [DEBUG] [app] Starting request: urn:request:60015a5f-14b1-496d-9b34-6489c231ba1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:22,637 [241] [DEBUG] [app] Starting request: urn:request:a4fc87d7-5b45-47b1-80ba-e87669ee99f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:22,637 [239] [DEBUG] [app] Ending request: urn:request:60015a5f-14b1-496d-9b34-6489c231ba1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:60015a5f-14b1-496d-9b34-6489c231ba1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:22,638 [241] [DEBUG] [app] Ending request: urn:request:a4fc87d7-5b45-47b1-80ba-e87669ee99f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4fc87d7-5b45-47b1-80ba-e87669ee99f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:22,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:22,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:22,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:22,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:22,649 [244] [DEBUG] [app] Ending request: urn:request:8f06f73b-7f83-40e8-972f-e5501634a085 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f06f73b-7f83-40e8-972f-e5501634a085', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:22,649 [243] [DEBUG] [app] Ending request: urn:request:b026b1d0-8963-4532-870f-d513e772cdf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b026b1d0-8963-4532-870f-d513e772cdf5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:22,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:22,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:37:22,650 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) securityworker stdout | 2023-01-17 05:37:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:39.230038+00:00 (in 16.392246 seconds) securityworker stdout | 2023-01-17 05:37:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:52 UTC)" (scheduled at 2023-01-17 05:37:22.837365+00:00) securityworker stdout | 2023-01-17 05:37:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:37:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:37:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:48.830274+00:00 (in 24.188025 seconds) exportactionlogsworker stdout | 2023-01-17 05:37:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:24 UTC)" (scheduled at 2023-01-17 05:37:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:37:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:37:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:37:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:37:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:20.242198+00:00 (in 54.907569 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:37:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:25 UTC)" (scheduled at 2023-01-17 05:37:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:37:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:37:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:37:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:37:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:57.226424+00:00 (in 29.999545 seconds) buildlogsarchiver stdout | 2023-01-17 05:37:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:57 UTC)" (scheduled at 2023-01-17 05:37:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:37:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 37, 27, 227244), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:37:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:37:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:37:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:38.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 05:37:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:38 UTC)" (scheduled at 2023-01-17 05:37:28.735066+00:00) notificationworker stdout | 2023-01-17 05:37:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 28, 736020), True, datetime.datetime(2023, 1, 17, 5, 37, 28, 736020), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:30,806 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:37:30,964 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:37:31,206 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:37:31,547 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:31,556 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:31,556 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:31,562 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:37:32,053 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:37:32,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:37:33,330 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:37:34,064 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:37:34,169 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:37:34,559 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:37:35,452 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:37:35,478 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:37:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:37:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:05.848675+00:00 (in 29.999505 seconds) gcworker stdout | 2023-01-17 05:37:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:05 UTC)" (scheduled at 2023-01-17 05:37:35.848675+00:00) gcworker stdout | 2023-01-17 05:37:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724255849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:37:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:37:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:37:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:37:36,021 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:37:36,023 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:37:36,026 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:37:36,354 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:37:37,124 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:37:37,139 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:37:37,626 [244] [DEBUG] [app] Starting request: urn:request:93a81b49-1c11-4126-83c4-66b012eddd9b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:37,626 [243] [DEBUG] [app] Starting request: urn:request:f1e314b2-8822-4d7e-b9fa-83597df58946 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:37,631 [232] [DEBUG] [app] Starting request: urn:request:dfed8bb6-fd5b-419f-9f29-68df19f0a275 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:37,631 [242] [DEBUG] [app] Starting request: urn:request:c476945d-8b16-4702-9746-3005ec96cdcd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:37,631 [232] [DEBUG] [app] Ending request: urn:request:dfed8bb6-fd5b-419f-9f29-68df19f0a275 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dfed8bb6-fd5b-419f-9f29-68df19f0a275', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:37:37,631 [242] [DEBUG] [app] Ending request: urn:request:c476945d-8b16-4702-9746-3005ec96cdcd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c476945d-8b16-4702-9746-3005ec96cdcd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:37:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:37:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:37,635 [243] [DEBUG] [app] Starting request: urn:request:e5b2e849-94a6-4186-b5c0-3efae9f0ca76 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:37,636 [243] [DEBUG] [app] Ending request: urn:request:e5b2e849-94a6-4186-b5c0-3efae9f0ca76 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5b2e849-94a6-4186-b5c0-3efae9f0ca76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:37,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:37,637 [239] [DEBUG] [app] Starting request: urn:request:19951ac8-dc33-4e62-8c46-8ff6a80fa378 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:37,638 [239] [DEBUG] [app] Ending request: urn:request:19951ac8-dc33-4e62-8c46-8ff6a80fa378 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19951ac8-dc33-4e62-8c46-8ff6a80fa378', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:37,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:37:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:37,646 [244] [DEBUG] [app] Ending request: urn:request:93a81b49-1c11-4126-83c4-66b012eddd9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93a81b49-1c11-4126-83c4-66b012eddd9b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:37,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:37,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.021 47 0.022) gunicorn-web stdout | 2023-01-17 05:37:37,648 [243] [DEBUG] [app] Ending request: urn:request:f1e314b2-8822-4d7e-b9fa-83597df58946 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1e314b2-8822-4d7e-b9fa-83597df58946', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:37:38,378 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:37:38,388 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:37:38,392 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:37:38,396 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:37:38,398 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:37:38,401 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:37:38,407 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:37:38,409 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:37:38,415 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:37:38,428 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:37:38,564 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:37:38,567 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:37:38,569 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:37:38,572 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:37:38,574 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:37:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:43.935629+00:00 (in 5.200078 seconds) notificationworker stdout | 2023-01-17 05:37:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:48 UTC)" (scheduled at 2023-01-17 05:37:38.735066+00:00) notificationworker stdout | 2023-01-17 05:37:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 38, 735951), True, datetime.datetime(2023, 1, 17, 5, 37, 38, 735951), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:38,933 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} repositorygcworker stdout | 2023-01-17 05:37:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:37:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:48.738458+00:00 (in 9.707159 seconds) repositorygcworker stdout | 2023-01-17 05:37:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:39 UTC)" (scheduled at 2023-01-17 05:37:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:37:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:37:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 39, 31761), True, datetime.datetime(2023, 1, 17, 5, 37, 39, 31761), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:37:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:37:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:37:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:52.837365+00:00 (in 13.606870 seconds) securityworker stdout | 2023-01-17 05:37:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:09 UTC)" (scheduled at 2023-01-17 05:37:39.230038+00:00) securityworker stdout | 2023-01-17 05:37:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:37:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:37:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:37:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:37:39,755 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:37:39,802 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:37:39,804 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:37:39,818 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:37:41,572 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:41,581 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:41,581 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:41,587 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:48.735066+00:00 (in 4.798960 seconds) notificationworker stdout | 2023-01-17 05:37:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:43 UTC)" (scheduled at 2023-01-17 05:37:43.935629+00:00) notificationworker stdout | 2023-01-17 05:37:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:37:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:37:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:37:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:59.337413+00:00 (in 11.404443 seconds) namespacegcworker stdout | 2023-01-17 05:37:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:47 UTC)" (scheduled at 2023-01-17 05:37:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:37:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:37:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:58.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 05:37:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:58 UTC)" (scheduled at 2023-01-17 05:37:48.735066+00:00) notificationworker stdout | 2023-01-17 05:37:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 48, 736001), True, datetime.datetime(2023, 1, 17, 5, 37, 48, 736001), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:37:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:37:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:39.030814+00:00 (in 50.291897 seconds) repositorygcworker stdout | 2023-01-17 05:37:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:48 UTC)" (scheduled at 2023-01-17 05:37:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:37:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:37:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:37:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:37:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:24.641816+00:00 (in 35.811054 seconds) exportactionlogsworker stdout | 2023-01-17 05:37:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:48 UTC)" (scheduled at 2023-01-17 05:37:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:37:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:37:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 48, 831176), True, datetime.datetime(2023, 1, 17, 5, 37, 48, 831176), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:37:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:37:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:37:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:48 UTC)" executed successfully builder stdout | 2023-01-17 05:37:51,597 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:51,606 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:51,606 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:51,612 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:52,625 [239] [DEBUG] [app] Starting request: urn:request:c0d3d740-bc9c-4e75-a4ad-251fd6953e34 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:52,626 [244] [DEBUG] [app] Starting request: urn:request:32622cfa-f261-4daa-a632-7ba4d709d45d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:37:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:52,631 [232] [DEBUG] [app] Starting request: urn:request:0eb9b6d5-084e-4541-8eec-4fc7bfcde1ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:52,631 [237] [DEBUG] [app] Starting request: urn:request:ee017258-d698-49e8-8ae4-0f014f14ebdd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:52,631 [232] [DEBUG] [app] Ending request: urn:request:0eb9b6d5-084e-4541-8eec-4fc7bfcde1ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0eb9b6d5-084e-4541-8eec-4fc7bfcde1ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:37:52,632 [237] [DEBUG] [app] Ending request: urn:request:ee017258-d698-49e8-8ae4-0f014f14ebdd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee017258-d698-49e8-8ae4-0f014f14ebdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:37:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:37:52,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:52,636 [241] [DEBUG] [app] Starting request: urn:request:b4c34a94-d5dc-4556-bde8-c5df7f1104fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:52,636 [243] [DEBUG] [app] Starting request: urn:request:167956c2-9f75-4663-bbbb-ba6f6acf77c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:52,636 [241] [DEBUG] [app] Ending request: urn:request:b4c34a94-d5dc-4556-bde8-c5df7f1104fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4c34a94-d5dc-4556-bde8-c5df7f1104fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:52,636 [243] [DEBUG] [app] Ending request: urn:request:167956c2-9f75-4663-bbbb-ba6f6acf77c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:167956c2-9f75-4663-bbbb-ba6f6acf77c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:52,647 [239] [DEBUG] [app] Ending request: urn:request:c0d3d740-bc9c-4e75-a4ad-251fd6953e34 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0d3d740-bc9c-4e75-a4ad-251fd6953e34', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:52,647 [244] [DEBUG] [app] Ending request: urn:request:32622cfa-f261-4daa-a632-7ba4d709d45d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:32622cfa-f261-4daa-a632-7ba4d709d45d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:37:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:37:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:09.230038+00:00 (in 16.392209 seconds) securityworker stdout | 2023-01-17 05:37:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:22 UTC)" (scheduled at 2023-01-17 05:37:52.837365+00:00) securityworker stdout | 2023-01-17 05:37:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:37:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:37:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:27.226424+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:37:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:27 UTC)" (scheduled at 2023-01-17 05:37:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:37:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 37, 57, 227279), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:37:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:37:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:37:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:08.735066+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:37:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:08 UTC)" (scheduled at 2023-01-17 05:37:58.735066+00:00) notificationworker stdout | 2023-01-17 05:37:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 58, 735928), True, datetime.datetime(2023, 1, 17, 5, 37, 58, 735928), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:37:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:37:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:47.932490+00:00 (in 48.594619 seconds) namespacegcworker stdout | 2023-01-17 05:37:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:59 UTC)" (scheduled at 2023-01-17 05:37:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:37:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:37:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 59, 338340), True, datetime.datetime(2023, 1, 17, 5, 37, 59, 338340), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:37:59,349 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:37:59,350 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:37:59,350 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:00,820 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:38:01,000 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:38:01,241 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:38:01,622 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:01,635 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:01,635 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:01,641 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:38:02,070 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:38:02,114 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:38:03,358 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:38:04,085 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:38:04,201 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:38:04,580 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:38:05,488 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:38:05,513 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:38:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:38:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:35.848675+00:00 (in 29.999496 seconds) gcworker stdout | 2023-01-17 05:38:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:35 UTC)" (scheduled at 2023-01-17 05:38:05.848675+00:00) gcworker stdout | 2023-01-17 05:38:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724285849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:38:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:38:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:38:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:38:06,029 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:38:06,031 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:38:06,033 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:38:06,375 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:38:07,157 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:38:07,174 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:38:07,626 [241] [DEBUG] [app] Starting request: urn:request:48324fad-e7e4-4412-bbef-8235117c1308 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:07,626 [244] [DEBUG] [app] Starting request: urn:request:33d59199-b413-4a70-a5e1-25c3edf79faf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:07,630 [232] [DEBUG] [app] Starting request: urn:request:ca9664d7-1f08-4f2b-8798-9c5db768f1d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,630 [242] [DEBUG] [app] Starting request: urn:request:c08dfea1-cacf-47fe-9f55-f96c7b6151f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,630 [232] [DEBUG] [app] Ending request: urn:request:ca9664d7-1f08-4f2b-8798-9c5db768f1d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ca9664d7-1f08-4f2b-8798-9c5db768f1d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,631 [242] [DEBUG] [app] Ending request: urn:request:c08dfea1-cacf-47fe-9f55-f96c7b6151f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c08dfea1-cacf-47fe-9f55-f96c7b6151f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:38:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:38:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:07,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:07,635 [243] [DEBUG] [app] Starting request: urn:request:fd533a07-a1ee-432a-9af0-201c6e46375c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:07,635 [243] [DEBUG] [app] Ending request: urn:request:fd533a07-a1ee-432a-9af0-201c6e46375c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd533a07-a1ee-432a-9af0-201c6e46375c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:07,635 [239] [DEBUG] [app] Starting request: urn:request:0f2347eb-b551-483e-8030-960c023de254 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:07,636 [239] [DEBUG] [app] Ending request: urn:request:0f2347eb-b551-483e-8030-960c023de254 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0f2347eb-b551-483e-8030-960c023de254', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:07,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:38:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:07,647 [241] [DEBUG] [app] Ending request: urn:request:48324fad-e7e4-4412-bbef-8235117c1308 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48324fad-e7e4-4412-bbef-8235117c1308', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:07,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:38:07,648 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:07,653 [244] [DEBUG] [app] Ending request: urn:request:33d59199-b413-4a70-a5e1-25c3edf79faf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33d59199-b413-4a70-a5e1-25c3edf79faf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:07,653 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:07,653 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-registry stdout | 2023-01-17 05:38:08,386 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:38:08,397 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:38:08,399 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:38:08,404 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:38:08,406 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:38:08,412 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:38:08,415 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:38:08,417 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:38:08,422 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:38:08,459 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:38:08,570 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:38:08,573 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:38:08,576 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:38:08,578 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:38:08,580 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:38:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:18.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 05:38:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:18 UTC)" (scheduled at 2023-01-17 05:38:08.735066+00:00) notificationworker stdout | 2023-01-17 05:38:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 8, 735910), True, datetime.datetime(2023, 1, 17, 5, 38, 8, 735910), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:08,969 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:38:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:22.837365+00:00 (in 13.606924 seconds) securityworker stdout | 2023-01-17 05:38:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:39 UTC)" (scheduled at 2023-01-17 05:38:09.230038+00:00) securityworker stdout | 2023-01-17 05:38:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:38:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:38:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:38:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:38:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:38:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:38:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:38:09,773 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:38:09,814 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:38:09,834 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:38:09,836 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} builder stdout | 2023-01-17 05:38:11,651 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:11,660 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:11,660 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:11,666 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:28.735066+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 05:38:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:28 UTC)" (scheduled at 2023-01-17 05:38:18.735066+00:00) notificationworker stdout | 2023-01-17 05:38:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 18, 735937), True, datetime.datetime(2023, 1, 17, 5, 38, 18, 735937), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:38:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:38:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:25.334138+00:00 (in 5.091463 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:38:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:20 UTC)" (scheduled at 2023-01-17 05:38:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:38:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:38:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 20, 243096), True, datetime.datetime(2023, 1, 17, 5, 38, 20, 243096), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:38:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:38:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:38:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:20 UTC)" executed successfully builder stdout | 2023-01-17 05:38:21,674 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:21,683 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:21,683 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:21,689 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:22,625 [239] [DEBUG] [app] Starting request: urn:request:8804b329-1aa6-448c-bd92-e7243de64eeb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,625 [244] [DEBUG] [app] Starting request: urn:request:0c9a43aa-e5eb-42d8-9f32-1f06a5dfee43 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:22,630 [232] [DEBUG] [app] Starting request: urn:request:8376efe0-4138-4d7b-b794-3b8f568b2f0e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,630 [242] [DEBUG] [app] Starting request: urn:request:fc638aee-f32a-4fb6-913d-9c56c3426f21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,631 [232] [DEBUG] [app] Ending request: urn:request:8376efe0-4138-4d7b-b794-3b8f568b2f0e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8376efe0-4138-4d7b-b794-3b8f568b2f0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,631 [242] [DEBUG] [app] Ending request: urn:request:fc638aee-f32a-4fb6-913d-9c56c3426f21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc638aee-f32a-4fb6-913d-9c56c3426f21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:38:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:38:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:22,636 [243] [DEBUG] [app] Starting request: urn:request:3b9d18fd-c770-4209-ac6b-37438b176209 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,636 [241] [DEBUG] [app] Starting request: urn:request:4cb16f04-7617-4084-8631-c5f96ced1b5a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,636 [243] [DEBUG] [app] Ending request: urn:request:3b9d18fd-c770-4209-ac6b-37438b176209 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b9d18fd-c770-4209-ac6b-37438b176209', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:22,636 [241] [DEBUG] [app] Ending request: urn:request:4cb16f04-7617-4084-8631-c5f96ced1b5a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4cb16f04-7617-4084-8631-c5f96ced1b5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:22,647 [239] [DEBUG] [app] Ending request: urn:request:8804b329-1aa6-448c-bd92-e7243de64eeb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8804b329-1aa6-448c-bd92-e7243de64eeb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:22,647 [244] [DEBUG] [app] Ending request: urn:request:0c9a43aa-e5eb-42d8-9f32-1f06a5dfee43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c9a43aa-e5eb-42d8-9f32-1f06a5dfee43', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:38:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:38:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:39.230038+00:00 (in 16.392245 seconds) securityworker stdout | 2023-01-17 05:38:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:52 UTC)" (scheduled at 2023-01-17 05:38:22.837365+00:00) securityworker stdout | 2023-01-17 05:38:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:38:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:38:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:48.830274+00:00 (in 24.187971 seconds) exportactionlogsworker stdout | 2023-01-17 05:38:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:24 UTC)" (scheduled at 2023-01-17 05:38:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:38:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:38:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:38:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:38:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:20.242198+00:00 (in 54.907566 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:38:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:25 UTC)" (scheduled at 2023-01-17 05:38:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:38:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:38:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:38:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:38:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:57.226424+00:00 (in 29.999553 seconds) buildlogsarchiver stdout | 2023-01-17 05:38:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:57 UTC)" (scheduled at 2023-01-17 05:38:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:38:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 38, 27, 227226), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:38:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:38:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:38:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:38.735066+00:00 (in 9.999494 seconds) notificationworker stdout | 2023-01-17 05:38:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:38 UTC)" (scheduled at 2023-01-17 05:38:28.735066+00:00) notificationworker stdout | 2023-01-17 05:38:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 28, 736032), True, datetime.datetime(2023, 1, 17, 5, 38, 28, 736032), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:30,829 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:38:31,036 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:38:31,254 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:38:31,699 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:31,714 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:31,714 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:31,719 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:38:32,079 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:38:32,149 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:38:33,382 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:38:34,121 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:38:34,236 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:38:34,590 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:38:35,524 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:38:35,547 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:38:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:38:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:05.848675+00:00 (in 29.999515 seconds) gcworker stdout | 2023-01-17 05:38:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:05 UTC)" (scheduled at 2023-01-17 05:38:35.848675+00:00) gcworker stdout | 2023-01-17 05:38:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724315849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:38:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:38:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:38:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:38:36,037 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:38:36,039 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:38:36,041 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:38:36,410 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:38:37,192 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:38:37,209 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:38:37,625 [239] [DEBUG] [app] Starting request: urn:request:44a9c299-faf4-45f3-9b8e-d6dcc9a381ba (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:37,626 [243] [DEBUG] [app] Starting request: urn:request:5ff7dba6-03ab-4acc-834f-0f22a6935605 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:37,630 [237] [DEBUG] [app] Starting request: urn:request:ee7d1f42-034d-40a4-926e-2db5db3d2793 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:37,631 [242] [DEBUG] [app] Starting request: urn:request:1e07a3a6-f0ad-4419-b425-2c56f808478e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:37,631 [237] [DEBUG] [app] Ending request: urn:request:ee7d1f42-034d-40a4-926e-2db5db3d2793 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee7d1f42-034d-40a4-926e-2db5db3d2793', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:38:37,631 [242] [DEBUG] [app] Ending request: urn:request:1e07a3a6-f0ad-4419-b425-2c56f808478e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e07a3a6-f0ad-4419-b425-2c56f808478e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:38:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:37,636 [244] [DEBUG] [app] Starting request: urn:request:6908383f-f1d3-4ea4-ba17-cabcc67415fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:37,637 [244] [DEBUG] [app] Ending request: urn:request:6908383f-f1d3-4ea4-ba17-cabcc67415fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6908383f-f1d3-4ea4-ba17-cabcc67415fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:37,637 [241] [DEBUG] [app] Starting request: urn:request:91ed974a-def8-4467-a709-edefded51684 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:38:37,637 [241] [DEBUG] [app] Ending request: urn:request:91ed974a-def8-4467-a709-edefded51684 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91ed974a-def8-4467-a709-edefded51684', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:37,649 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:37,650 [239] [DEBUG] [app] Ending request: urn:request:44a9c299-faf4-45f3-9b8e-d6dcc9a381ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44a9c299-faf4-45f3-9b8e-d6dcc9a381ba', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:37,650 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) gunicorn-web stdout | 2023-01-17 05:38:37,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:38:37,654 [243] [DEBUG] [app] Ending request: urn:request:5ff7dba6-03ab-4acc-834f-0f22a6935605 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ff7dba6-03ab-4acc-834f-0f22a6935605', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:37,654 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:37,654 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-registry stdout | 2023-01-17 05:38:38,396 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:38:38,404 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:38:38,408 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:38:38,412 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:38:38,414 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:38:38,421 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:38:38,424 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:38:38,427 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:38:38,429 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:38:38,475 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:38:38,577 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:38:38,579 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:38:38,582 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:38:38,586 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:38:38,588 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:38:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:43.935629+00:00 (in 5.200108 seconds) notificationworker stdout | 2023-01-17 05:38:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:48 UTC)" (scheduled at 2023-01-17 05:38:38.735066+00:00) notificationworker stdout | 2023-01-17 05:38:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 38, 735920), True, datetime.datetime(2023, 1, 17, 5, 38, 38, 735920), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:39,005 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} repositorygcworker stdout | 2023-01-17 05:38:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:38:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:48.738458+00:00 (in 9.707223 seconds) repositorygcworker stdout | 2023-01-17 05:38:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:39 UTC)" (scheduled at 2023-01-17 05:38:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:38:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:38:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 39, 31672), True, datetime.datetime(2023, 1, 17, 5, 38, 39, 31672), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:38:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:38:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:38:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:52.837365+00:00 (in 13.606879 seconds) securityworker stdout | 2023-01-17 05:38:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:09 UTC)" (scheduled at 2023-01-17 05:38:39.230038+00:00) securityworker stdout | 2023-01-17 05:38:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:38:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:38:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:38:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:38:39,808 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:38:39,843 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:38:39,845 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:38:39,847 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:38:41,729 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:41,739 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:41,739 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:41,745 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:48.735066+00:00 (in 4.798992 seconds) notificationworker stdout | 2023-01-17 05:38:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:43 UTC)" (scheduled at 2023-01-17 05:38:43.935629+00:00) notificationworker stdout | 2023-01-17 05:38:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:38:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:38:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:38:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:59.337413+00:00 (in 11.404469 seconds) namespacegcworker stdout | 2023-01-17 05:38:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:47 UTC)" (scheduled at 2023-01-17 05:38:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:38:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:38:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:56.935055+00:00 (in 8.199537 seconds) notificationworker stdout | 2023-01-17 05:38:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:58 UTC)" (scheduled at 2023-01-17 05:38:48.735066+00:00) notificationworker stdout | 2023-01-17 05:38:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 48, 735927), True, datetime.datetime(2023, 1, 17, 5, 38, 48, 735927), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:38:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:38:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:39.030814+00:00 (in 50.291871 seconds) repositorygcworker stdout | 2023-01-17 05:38:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:48 UTC)" (scheduled at 2023-01-17 05:38:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:38:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:38:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:38:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:38:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:24.641816+00:00 (in 35.811090 seconds) exportactionlogsworker stdout | 2023-01-17 05:38:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:48 UTC)" (scheduled at 2023-01-17 05:38:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:38:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:38:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 48, 831104), True, datetime.datetime(2023, 1, 17, 5, 38, 48, 831104), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:38:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:38:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:38:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:48 UTC)" executed successfully builder stdout | 2023-01-17 05:38:51,755 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:51,764 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:51,764 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:51,770 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:52,625 [244] [DEBUG] [app] Starting request: urn:request:99a20acf-eb21-4f73-8377-35173565be5f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:52,625 [239] [DEBUG] [app] Starting request: urn:request:6beca6f5-69db-48e8-8659-f20ab105ec97 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:38:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:52,630 [237] [DEBUG] [app] Starting request: urn:request:27b6db95-cf64-4c88-a5e6-4c526a8701f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:52,630 [235] [DEBUG] [app] Starting request: urn:request:32aadffb-b2ef-496e-a9e0-7aa28ff8c95f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:52,630 [237] [DEBUG] [app] Ending request: urn:request:27b6db95-cf64-4c88-a5e6-4c526a8701f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27b6db95-cf64-4c88-a5e6-4c526a8701f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:38:52,631 [235] [DEBUG] [app] Ending request: urn:request:32aadffb-b2ef-496e-a9e0-7aa28ff8c95f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:32aadffb-b2ef-496e-a9e0-7aa28ff8c95f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:38:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:38:52,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:52,635 [243] [DEBUG] [app] Starting request: urn:request:3f11c759-6118-4a16-bcf9-8d69765afaff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:52,636 [243] [DEBUG] [app] Ending request: urn:request:3f11c759-6118-4a16-bcf9-8d69765afaff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f11c759-6118-4a16-bcf9-8d69765afaff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:52,636 [241] [DEBUG] [app] Starting request: urn:request:434746d4-7e00-479e-8dd8-7ba8516912ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:52,636 [241] [DEBUG] [app] Ending request: urn:request:434746d4-7e00-479e-8dd8-7ba8516912ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:434746d4-7e00-479e-8dd8-7ba8516912ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:38:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:52,647 [244] [DEBUG] [app] Ending request: urn:request:99a20acf-eb21-4f73-8377-35173565be5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99a20acf-eb21-4f73-8377-35173565be5f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:52,647 [239] [DEBUG] [app] Ending request: urn:request:6beca6f5-69db-48e8-8659-f20ab105ec97 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6beca6f5-69db-48e8-8659-f20ab105ec97', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:52,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:38:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:38:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:38:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:09.230038+00:00 (in 16.392224 seconds) securityworker stdout | 2023-01-17 05:38:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:22 UTC)" (scheduled at 2023-01-17 05:38:52.837365+00:00) securityworker stdout | 2023-01-17 05:38:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:52,844 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:58.735066+00:00 (in 1.799544 seconds) notificationworker stdout | 2023-01-17 05:38:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:43:56 UTC)" (scheduled at 2023-01-17 05:38:56.935055+00:00) notificationworker stdout | 2023-01-17 05:38:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 38, 56, 935883), 'notification/%']) notificationworker stdout | 2023-01-17 05:38:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 38, 56, 935883), True, datetime.datetime(2023, 1, 17, 5, 38, 56, 935883), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:38:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 38, 56, 935883), True, datetime.datetime(2023, 1, 17, 5, 38, 56, 935883), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 38, 56, 935883), 'notification/%']) notificationworker stdout | 2023-01-17 05:38:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:43:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:38:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:38:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:27.226424+00:00 (in 29.999535 seconds) buildlogsarchiver stdout | 2023-01-17 05:38:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:27 UTC)" (scheduled at 2023-01-17 05:38:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:38:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 38, 57, 227259), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:38:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:38:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:38:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:08.735066+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 05:38:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:08 UTC)" (scheduled at 2023-01-17 05:38:58.735066+00:00) notificationworker stdout | 2023-01-17 05:38:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 58, 735936), True, datetime.datetime(2023, 1, 17, 5, 38, 58, 735936), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:38:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:38:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:15.931994+00:00 (in 16.594114 seconds) namespacegcworker stdout | 2023-01-17 05:38:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:59 UTC)" (scheduled at 2023-01-17 05:38:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:38:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:38:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 59, 338279), True, datetime.datetime(2023, 1, 17, 5, 38, 59, 338279), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:38:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:38:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:38:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:39:00,865 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:39:01,069 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:39:01,289 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:39:01,775 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:01,784 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:01,784 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:01,790 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:39:02,113 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:39:02,184 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:39:03,418 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:39:04,157 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:39:04,263 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:39:04,597 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:39:05,549 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:39:05,570 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:39:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:39:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:35.848675+00:00 (in 29.999488 seconds) gcworker stdout | 2023-01-17 05:39:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:35 UTC)" (scheduled at 2023-01-17 05:39:05.848675+00:00) gcworker stdout | 2023-01-17 05:39:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724345849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:39:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:39:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:39:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:39:06,043 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:39:06,046 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:39:06,048 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:39:06,437 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:39:07,225 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:39:07,228 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:39:07,625 [239] [DEBUG] [app] Starting request: urn:request:c3313193-a563-4372-8383-863003902c55 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:07,625 [241] [DEBUG] [app] Starting request: urn:request:d0f6499d-b2da-454b-a7ba-4e6c99ecf644 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:07,630 [242] [DEBUG] [app] Starting request: urn:request:51db7c6f-de3d-472c-8df7-110ef643ec0e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:07,630 [237] [DEBUG] [app] Starting request: urn:request:f03c1d10-d263-4524-9dce-33cb39549387 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:07,630 [242] [DEBUG] [app] Ending request: urn:request:51db7c6f-de3d-472c-8df7-110ef643ec0e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51db7c6f-de3d-472c-8df7-110ef643ec0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:39:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:39:07,631 [237] [DEBUG] [app] Ending request: urn:request:f03c1d10-d263-4524-9dce-33cb39549387 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f03c1d10-d263-4524-9dce-33cb39549387', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:39:07,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:39:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:07,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:07,635 [244] [DEBUG] [app] Starting request: urn:request:d536007c-4634-4a6b-9264-b30e2add8f17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:07,635 [239] [DEBUG] [app] Starting request: urn:request:4bc4a061-b3be-4943-9527-e6ae3637b9cb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:07,635 [239] [DEBUG] [app] Ending request: urn:request:4bc4a061-b3be-4943-9527-e6ae3637b9cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4bc4a061-b3be-4943-9527-e6ae3637b9cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:07,635 [244] [DEBUG] [app] Ending request: urn:request:d536007c-4634-4a6b-9264-b30e2add8f17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d536007c-4634-4a6b-9264-b30e2add8f17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:07,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:07,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:39:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:07,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:07,647 [239] [DEBUG] [app] Ending request: urn:request:c3313193-a563-4372-8383-863003902c55 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3313193-a563-4372-8383-863003902c55', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:07,647 [241] [DEBUG] [app] Ending request: urn:request:d0f6499d-b2da-454b-a7ba-4e6c99ecf644 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d0f6499d-b2da-454b-a7ba-4e6c99ecf644', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:39:07,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:39:08,407 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:39:08,410 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:39:08,415 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:39:08,418 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:39:08,421 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:39:08,429 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:39:08,432 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:39:08,434 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:39:08,436 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:39:08,494 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:39:08,584 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:39:08,587 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:39:08,589 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:39:08,592 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:39:08,594 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:39:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:18.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 05:39:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:18 UTC)" (scheduled at 2023-01-17 05:39:08.735066+00:00) notificationworker stdout | 2023-01-17 05:39:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 8, 735885), True, datetime.datetime(2023, 1, 17, 5, 39, 8, 735885), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:09,037 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:39:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:22.837365+00:00 (in 13.606872 seconds) securityworker stdout | 2023-01-17 05:39:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:39 UTC)" (scheduled at 2023-01-17 05:39:09.230038+00:00) securityworker stdout | 2023-01-17 05:39:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:39:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:39:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:39:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:39:09,831 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:39:09,878 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:39:09,881 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:39:09,883 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:39:11,800 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:11,809 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:11,809 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:11,815 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:39:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:39:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:47.932490+00:00 (in 31.999994 seconds) namespacegcworker stdout | 2023-01-17 05:39:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:44:15 UTC)" (scheduled at 2023-01-17 05:39:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:39:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 39, 15, 932882), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:39:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 39, 15, 932882), True, datetime.datetime(2023, 1, 17, 5, 39, 15, 932882), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:39:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 39, 15, 932882), True, datetime.datetime(2023, 1, 17, 5, 39, 15, 932882), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 39, 15, 932882), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:39:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:39:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:44:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:28.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 05:39:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:28 UTC)" (scheduled at 2023-01-17 05:39:18.735066+00:00) notificationworker stdout | 2023-01-17 05:39:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 18, 735924), True, datetime.datetime(2023, 1, 17, 5, 39, 18, 735924), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:39:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:39:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:25.334138+00:00 (in 5.091461 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:39:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:20 UTC)" (scheduled at 2023-01-17 05:39:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:39:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:39:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 20, 243097), True, datetime.datetime(2023, 1, 17, 5, 39, 20, 243097), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:39:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:39:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:39:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:20 UTC)" executed successfully builder stdout | 2023-01-17 05:39:21,825 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:21,834 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:21,834 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:21,840 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:22,625 [239] [DEBUG] [app] Starting request: urn:request:fa387bef-0b2b-4916-a33a-fdb6263eaf0e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:22,625 [244] [DEBUG] [app] Starting request: urn:request:fdd7f6b8-5cb9-4dc2-8908-62c8cc3ea9ac (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:22,631 [242] [DEBUG] [app] Starting request: urn:request:f42249ca-c17c-4d9f-b721-ded06e4dba58 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:22,631 [242] [DEBUG] [app] Ending request: urn:request:f42249ca-c17c-4d9f-b721-ded06e4dba58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f42249ca-c17c-4d9f-b721-ded06e4dba58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:39:22,631 [237] [DEBUG] [app] Starting request: urn:request:6d6a8958-f1a1-4956-9d71-0bfc9f05e48d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:39:22,632 [237] [DEBUG] [app] Ending request: urn:request:6d6a8958-f1a1-4956-9d71-0bfc9f05e48d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d6a8958-f1a1-4956-9d71-0bfc9f05e48d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:39:22,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:39:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:22,636 [243] [DEBUG] [app] Starting request: urn:request:ca69bcb3-afe2-4d51-bf65-a4b45c4a7609 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:22,636 [241] [DEBUG] [app] Starting request: urn:request:f4741b57-557f-496f-ba53-3bf85d29991b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:22,636 [243] [DEBUG] [app] Ending request: urn:request:ca69bcb3-afe2-4d51-bf65-a4b45c4a7609 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca69bcb3-afe2-4d51-bf65-a4b45c4a7609', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:22,637 [241] [DEBUG] [app] Ending request: urn:request:f4741b57-557f-496f-ba53-3bf85d29991b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4741b57-557f-496f-ba53-3bf85d29991b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:39:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:39:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:22,648 [244] [DEBUG] [app] Ending request: urn:request:fdd7f6b8-5cb9-4dc2-8908-62c8cc3ea9ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fdd7f6b8-5cb9-4dc2-8908-62c8cc3ea9ac', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:22,648 [239] [DEBUG] [app] Ending request: urn:request:fa387bef-0b2b-4916-a33a-fdb6263eaf0e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fa387bef-0b2b-4916-a33a-fdb6263eaf0e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:39:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:39:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:39.230038+00:00 (in 16.392246 seconds) securityworker stdout | 2023-01-17 05:39:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:52 UTC)" (scheduled at 2023-01-17 05:39:22.837365+00:00) securityworker stdout | 2023-01-17 05:39:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:39:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:39:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:48.830274+00:00 (in 24.187984 seconds) exportactionlogsworker stdout | 2023-01-17 05:39:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:24 UTC)" (scheduled at 2023-01-17 05:39:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:39:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:39:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:39:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:39:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:20.242198+00:00 (in 54.907606 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:39:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:25 UTC)" (scheduled at 2023-01-17 05:39:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:39:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:39:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:39:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:39:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:57.226424+00:00 (in 29.999552 seconds) buildlogsarchiver stdout | 2023-01-17 05:39:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:57 UTC)" (scheduled at 2023-01-17 05:39:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:39:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 39, 27, 227203), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:39:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:39:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:39:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:38.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 05:39:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:38 UTC)" (scheduled at 2023-01-17 05:39:28.735066+00:00) notificationworker stdout | 2023-01-17 05:39:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 28, 735894), True, datetime.datetime(2023, 1, 17, 5, 39, 28, 735894), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:39:30,901 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:39:31,083 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:39:31,299 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:39:31,850 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:31,861 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:31,861 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:31,866 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:39:32,123 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:39:32,194 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:39:33,454 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:39:34,165 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:39:34,274 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:39:34,617 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:39:35,584 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:39:35,587 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:39:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:39:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:05.848675+00:00 (in 29.999536 seconds) gcworker stdout | 2023-01-17 05:39:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:05 UTC)" (scheduled at 2023-01-17 05:39:35.848675+00:00) gcworker stdout | 2023-01-17 05:39:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724375849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:39:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:39:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:39:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:39:36,051 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:39:36,054 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:39:36,056 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:39:36,449 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:39:37,242 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:39:37,259 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:39:37,625 [243] [DEBUG] [app] Starting request: urn:request:cd375e9b-4439-4057-a944-1ae3bf192739 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:37,625 [244] [DEBUG] [app] Starting request: urn:request:5e8a39d4-fffc-473b-8637-3066f8882052 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:37,630 [232] [DEBUG] [app] Starting request: urn:request:25a07429-f8f3-4ebf-8d5d-1a8696ef5370 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:37,631 [242] [DEBUG] [app] Starting request: urn:request:530b47b7-2eff-44ea-9102-a99323f2ed1b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:37,631 [232] [DEBUG] [app] Ending request: urn:request:25a07429-f8f3-4ebf-8d5d-1a8696ef5370 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:25a07429-f8f3-4ebf-8d5d-1a8696ef5370', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:39:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:39:37,631 [242] [DEBUG] [app] Ending request: urn:request:530b47b7-2eff-44ea-9102-a99323f2ed1b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:530b47b7-2eff-44ea-9102-a99323f2ed1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:39:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:37,635 [239] [DEBUG] [app] Starting request: urn:request:9b2c9c92-cd95-4350-96de-3da5555b74c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:37,635 [243] [DEBUG] [app] Starting request: urn:request:ac83bb3d-e0ed-43c0-b4ed-505b281229c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:37,636 [239] [DEBUG] [app] Ending request: urn:request:9b2c9c92-cd95-4350-96de-3da5555b74c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b2c9c92-cd95-4350-96de-3da5555b74c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:37,636 [243] [DEBUG] [app] Ending request: urn:request:ac83bb3d-e0ed-43c0-b4ed-505b281229c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac83bb3d-e0ed-43c0-b4ed-505b281229c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.000) gunicorn-web stdout | 2023-01-17 05:39:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:37,647 [243] [DEBUG] [app] Ending request: urn:request:cd375e9b-4439-4057-a944-1ae3bf192739 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd375e9b-4439-4057-a944-1ae3bf192739', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:37,647 [244] [DEBUG] [app] Ending request: urn:request:5e8a39d4-fffc-473b-8637-3066f8882052 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5e8a39d4-fffc-473b-8637-3066f8882052', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:39:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:39:38,416 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:39:38,420 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:39:38,423 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:39:38,426 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:39:38,428 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:39:38,437 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:39:38,440 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:39:38,442 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:39:38,445 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:39:38,506 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:39:38,591 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:39:38,593 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:39:38,596 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:39:38,598 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:39:38,600 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:39:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:43.935629+00:00 (in 5.200132 seconds) notificationworker stdout | 2023-01-17 05:39:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:48 UTC)" (scheduled at 2023-01-17 05:39:38.735066+00:00) notificationworker stdout | 2023-01-17 05:39:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 38, 735894), True, datetime.datetime(2023, 1, 17, 5, 39, 38, 735894), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:39:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:39:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:48.738458+00:00 (in 9.707243 seconds) repositorygcworker stdout | 2023-01-17 05:39:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:39 UTC)" (scheduled at 2023-01-17 05:39:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:39:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:39:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 39, 31610), True, datetime.datetime(2023, 1, 17, 5, 39, 39, 31610), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:39:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:39:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:39:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:39,073 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:39:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:52.837365+00:00 (in 13.606917 seconds) securityworker stdout | 2023-01-17 05:39:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:09 UTC)" (scheduled at 2023-01-17 05:39:39.230038+00:00) securityworker stdout | 2023-01-17 05:39:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:39:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:39:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:39:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:39:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:39:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:39:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:39:39,867 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:39:39,907 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:39:39,913 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:39:39,915 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:39:41,869 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:41,879 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:41,879 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:41,885 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:48.735066+00:00 (in 4.798941 seconds) notificationworker stdout | 2023-01-17 05:39:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:43 UTC)" (scheduled at 2023-01-17 05:39:43.935629+00:00) notificationworker stdout | 2023-01-17 05:39:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:39:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:39:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:39:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:59.337413+00:00 (in 11.404411 seconds) namespacegcworker stdout | 2023-01-17 05:39:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:47 UTC)" (scheduled at 2023-01-17 05:39:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:39:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:39:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:58.735066+00:00 (in 9.999479 seconds) notificationworker stdout | 2023-01-17 05:39:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:58 UTC)" (scheduled at 2023-01-17 05:39:48.735066+00:00) notificationworker stdout | 2023-01-17 05:39:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 48, 736011), True, datetime.datetime(2023, 1, 17, 5, 39, 48, 736011), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:39:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:39:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:39.030814+00:00 (in 50.291941 seconds) repositorygcworker stdout | 2023-01-17 05:39:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:48 UTC)" (scheduled at 2023-01-17 05:39:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:39:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:39:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:48,748 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:48,748 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:48,748 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:39:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:39:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:24.641816+00:00 (in 35.810985 seconds) exportactionlogsworker stdout | 2023-01-17 05:39:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:48 UTC)" (scheduled at 2023-01-17 05:39:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:39:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:39:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 48, 831241), True, datetime.datetime(2023, 1, 17, 5, 39, 48, 831241), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:39:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:39:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:39:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:48 UTC)" executed successfully builder stdout | 2023-01-17 05:39:51,895 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:51,903 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:51,903 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:51,909 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:52,626 [243] [DEBUG] [app] Starting request: urn:request:7445959b-4b3f-48b9-b83a-df805bd38a2d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:52,626 [241] [DEBUG] [app] Starting request: urn:request:83545b0e-bc0d-4cf7-b313-72e4ec2cb47d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:39:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:52,631 [242] [DEBUG] [app] Starting request: urn:request:78c877e7-a03e-4758-859a-7c2a6ee3d761 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:52,631 [240] [DEBUG] [app] Starting request: urn:request:6a7588f9-7da7-4223-bf2a-1afaf2d42813 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:52,631 [242] [DEBUG] [app] Ending request: urn:request:78c877e7-a03e-4758-859a-7c2a6ee3d761 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78c877e7-a03e-4758-859a-7c2a6ee3d761', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:39:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:39:52,632 [240] [DEBUG] [app] Ending request: urn:request:6a7588f9-7da7-4223-bf2a-1afaf2d42813 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6a7588f9-7da7-4223-bf2a-1afaf2d42813', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:39:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:39:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:52,636 [239] [DEBUG] [app] Starting request: urn:request:32c5e44e-284e-4b5f-81a9-9204ccd5e548 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:52,636 [239] [DEBUG] [app] Ending request: urn:request:32c5e44e-284e-4b5f-81a9-9204ccd5e548 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32c5e44e-284e-4b5f-81a9-9204ccd5e548', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:52,636 [244] [DEBUG] [app] Starting request: urn:request:91a2a37c-cccd-46bb-b5af-0b5540cd914a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:39:52,637 [244] [DEBUG] [app] Ending request: urn:request:91a2a37c-cccd-46bb-b5af-0b5540cd914a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91a2a37c-cccd-46bb-b5af-0b5540cd914a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:39:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:52,648 [241] [DEBUG] [app] Ending request: urn:request:83545b0e-bc0d-4cf7-b313-72e4ec2cb47d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83545b0e-bc0d-4cf7-b313-72e4ec2cb47d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:52,648 [243] [DEBUG] [app] Ending request: urn:request:7445959b-4b3f-48b9-b83a-df805bd38a2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7445959b-4b3f-48b9-b83a-df805bd38a2d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:39:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:39:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:09.230038+00:00 (in 16.392229 seconds) securityworker stdout | 2023-01-17 05:39:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:22 UTC)" (scheduled at 2023-01-17 05:39:52.837365+00:00) securityworker stdout | 2023-01-17 05:39:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:39:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:39:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:27.226424+00:00 (in 29.999576 seconds) buildlogsarchiver stdout | 2023-01-17 05:39:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:27 UTC)" (scheduled at 2023-01-17 05:39:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:39:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 39, 57, 227196), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:39:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:39:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:39:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:08.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:39:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:08 UTC)" (scheduled at 2023-01-17 05:39:58.735066+00:00) notificationworker stdout | 2023-01-17 05:39:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 58, 735910), True, datetime.datetime(2023, 1, 17, 5, 39, 58, 735910), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:39:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:39:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:47.932490+00:00 (in 48.594579 seconds) namespacegcworker stdout | 2023-01-17 05:39:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:59 UTC)" (scheduled at 2023-01-17 05:39:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:39:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:39:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 59, 338336), True, datetime.datetime(2023, 1, 17, 5, 39, 59, 338336), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:39:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:39:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:39:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:40:00,918 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:40:01,119 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:40:01,334 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:40:01,913 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:01,922 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:01,922 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:01,928 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:40:02,132 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:40:02,207 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:40:03,469 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:40:04,194 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:40:04,293 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:40:04,653 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:40:05,620 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:40:05,623 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:40:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:40:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:35.848675+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 05:40:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:35 UTC)" (scheduled at 2023-01-17 05:40:05.848675+00:00) gcworker stdout | 2023-01-17 05:40:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724405849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:40:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:40:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:40:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:40:06,058 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:40:06,060 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:40:06,062 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:40:06,485 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:40:07,277 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:40:07,294 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:40:07,625 [239] [DEBUG] [app] Starting request: urn:request:4d489175-3992-4855-896a-59a971b4660e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:07,625 [244] [DEBUG] [app] Starting request: urn:request:155d82c6-c90e-45e2-9214-d27ef5c57be9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:07,630 [242] [DEBUG] [app] Starting request: urn:request:62ab1d09-a866-4434-82fc-2e0b985761dc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:07,630 [232] [DEBUG] [app] Starting request: urn:request:ef9cb72c-a816-46ee-9287-fc8e139cbc39 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:07,630 [242] [DEBUG] [app] Ending request: urn:request:62ab1d09-a866-4434-82fc-2e0b985761dc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:62ab1d09-a866-4434-82fc-2e0b985761dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:40:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:40:07,631 [232] [DEBUG] [app] Ending request: urn:request:ef9cb72c-a816-46ee-9287-fc8e139cbc39 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef9cb72c-a816-46ee-9287-fc8e139cbc39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:40:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:07,635 [244] [DEBUG] [app] Starting request: urn:request:7cd8b5a1-76ca-4512-b495-94e19bea8dd0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:07,635 [244] [DEBUG] [app] Ending request: urn:request:7cd8b5a1-76ca-4512-b495-94e19bea8dd0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7cd8b5a1-76ca-4512-b495-94e19bea8dd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:07,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:07,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:07,637 [243] [DEBUG] [app] Starting request: urn:request:f734fb4e-9033-46c6-806c-73ad85f44c33 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:07,637 [243] [DEBUG] [app] Ending request: urn:request:f734fb4e-9033-46c6-806c-73ad85f44c33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f734fb4e-9033-46c6-806c-73ad85f44c33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:07,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:07,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:07,647 [239] [DEBUG] [app] Ending request: urn:request:4d489175-3992-4855-896a-59a971b4660e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d489175-3992-4855-896a-59a971b4660e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:40:07,649 [244] [DEBUG] [app] Ending request: urn:request:155d82c6-c90e-45e2-9214-d27ef5c57be9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:155d82c6-c90e-45e2-9214-d27ef5c57be9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:07,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-registry stdout | 2023-01-17 05:40:08,423 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:40:08,429 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:40:08,431 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:40:08,434 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:40:08,436 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:40:08,443 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:40:08,446 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:40:08,452 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:40:08,455 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:40:08,518 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:40:08,597 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:40:08,601 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:40:08,604 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:40:08,606 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:40:08,609 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:40:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:18.735066+00:00 (in 9.999568 seconds) notificationworker stdout | 2023-01-17 05:40:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:18 UTC)" (scheduled at 2023-01-17 05:40:08.735066+00:00) notificationworker stdout | 2023-01-17 05:40:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 8, 735922), True, datetime.datetime(2023, 1, 17, 5, 40, 8, 735922), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:09,110 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:40:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:22.837365+00:00 (in 13.606853 seconds) securityworker stdout | 2023-01-17 05:40:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:39 UTC)" (scheduled at 2023-01-17 05:40:09.230038+00:00) securityworker stdout | 2023-01-17 05:40:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:40:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:40:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:40:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:40:09,903 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:40:09,934 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:40:09,945 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:40:09,950 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:40:11,938 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:11,947 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:11,947 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:11,953 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:28.735066+00:00 (in 9.999525 seconds) notificationworker stdout | 2023-01-17 05:40:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:28 UTC)" (scheduled at 2023-01-17 05:40:18.735066+00:00) notificationworker stdout | 2023-01-17 05:40:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 18, 735952), True, datetime.datetime(2023, 1, 17, 5, 40, 18, 735952), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:40:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:40:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:25.334138+00:00 (in 5.091501 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:40:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:20 UTC)" (scheduled at 2023-01-17 05:40:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:40:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:40:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 20, 243049), True, datetime.datetime(2023, 1, 17, 5, 40, 20, 243049), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:40:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:40:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:40:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:20 UTC)" executed successfully builder stdout | 2023-01-17 05:40:21,954 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:21,963 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:21,963 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:21,969 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:22,626 [239] [DEBUG] [app] Starting request: urn:request:eac19794-573e-4fdd-85bb-66398ac87ea5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:22,626 [244] [DEBUG] [app] Starting request: urn:request:5ce99f8c-b0b3-4e18-b132-94654b378687 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:22,630 [232] [DEBUG] [app] Starting request: urn:request:58c4a609-1d80-4c98-a941-192a239223ee (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:22,631 [235] [DEBUG] [app] Starting request: urn:request:fc91e3d1-b008-4d34-9ab9-188d37bf55dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:22,631 [232] [DEBUG] [app] Ending request: urn:request:58c4a609-1d80-4c98-a941-192a239223ee (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58c4a609-1d80-4c98-a941-192a239223ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:40:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:40:22,631 [235] [DEBUG] [app] Ending request: urn:request:fc91e3d1-b008-4d34-9ab9-188d37bf55dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc91e3d1-b008-4d34-9ab9-188d37bf55dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:40:22,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:40:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:22,635 [243] [DEBUG] [app] Starting request: urn:request:767001cb-8f76-4684-a842-cce62fd687ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:22,636 [241] [DEBUG] [app] Starting request: urn:request:f60bd7d0-fe83-449f-809b-2e190db26090 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:22,636 [243] [DEBUG] [app] Ending request: urn:request:767001cb-8f76-4684-a842-cce62fd687ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:767001cb-8f76-4684-a842-cce62fd687ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:40:22,636 [241] [DEBUG] [app] Ending request: urn:request:f60bd7d0-fe83-449f-809b-2e190db26090 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f60bd7d0-fe83-449f-809b-2e190db26090', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:22,647 [239] [DEBUG] [app] Ending request: urn:request:eac19794-573e-4fdd-85bb-66398ac87ea5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eac19794-573e-4fdd-85bb-66398ac87ea5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:22,647 [244] [DEBUG] [app] Ending request: urn:request:5ce99f8c-b0b3-4e18-b132-94654b378687 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ce99f8c-b0b3-4e18-b132-94654b378687', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:40:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:40:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:39.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 05:40:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:52 UTC)" (scheduled at 2023-01-17 05:40:22.837365+00:00) securityworker stdout | 2023-01-17 05:40:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:22,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:22,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:22,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:22,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:40:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:40:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:48.830274+00:00 (in 24.187974 seconds) exportactionlogsworker stdout | 2023-01-17 05:40:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:24 UTC)" (scheduled at 2023-01-17 05:40:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:40:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:40:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:40:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:40:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:20.242198+00:00 (in 54.907553 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:40:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:25 UTC)" (scheduled at 2023-01-17 05:40:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:40:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:40:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:40:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:40:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:57.226424+00:00 (in 29.999563 seconds) buildlogsarchiver stdout | 2023-01-17 05:40:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:57 UTC)" (scheduled at 2023-01-17 05:40:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:40:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 40, 27, 227246), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:40:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:40:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:40:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:38.735066+00:00 (in 9.999490 seconds) notificationworker stdout | 2023-01-17 05:40:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:38 UTC)" (scheduled at 2023-01-17 05:40:28.735066+00:00) notificationworker stdout | 2023-01-17 05:40:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 28, 736025), True, datetime.datetime(2023, 1, 17, 5, 40, 28, 736025), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:40:30,954 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:40:31,151 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:40:31,369 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:40:31,973 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:31,982 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:31,982 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:31,988 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:40:32,164 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:40:32,242 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:40:33,504 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:40:34,219 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:40:34,326 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:40:34,689 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositoryactioncounter stdout | 2023-01-17 05:40:35,645 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:40:35,655 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gcworker stdout | 2023-01-17 05:40:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:40:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:05.848675+00:00 (in 29.999567 seconds) gcworker stdout | 2023-01-17 05:40:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:05 UTC)" (scheduled at 2023-01-17 05:40:35.848675+00:00) gcworker stdout | 2023-01-17 05:40:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724435849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:40:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:40:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:40:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:40:36,065 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:40:36,067 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:40:36,069 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:40:36,519 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:40:37,314 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:40:37,329 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:40:37,625 [241] [DEBUG] [app] Starting request: urn:request:b374975d-4138-49bc-9aaf-8e3408637812 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:37,625 [239] [DEBUG] [app] Starting request: urn:request:5c05a087-426a-4ba5-88bf-50d919e116bf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:37,630 [242] [DEBUG] [app] Starting request: urn:request:19185583-c9c2-4e39-a289-2db1efd852d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:37,630 [240] [DEBUG] [app] Starting request: urn:request:d96fb0fb-2b1e-4981-9f06-159ceeb25738 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:37,630 [242] [DEBUG] [app] Ending request: urn:request:19185583-c9c2-4e39-a289-2db1efd852d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:19185583-c9c2-4e39-a289-2db1efd852d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:40:37,631 [240] [DEBUG] [app] Ending request: urn:request:d96fb0fb-2b1e-4981-9f06-159ceeb25738 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d96fb0fb-2b1e-4981-9f06-159ceeb25738', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:40:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:37,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:37,635 [244] [DEBUG] [app] Starting request: urn:request:e72c363f-e524-4eb0-a272-a30cac986318 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:37,635 [243] [DEBUG] [app] Starting request: urn:request:9dc825f7-9aad-4276-a216-38294d0058de (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:37,635 [244] [DEBUG] [app] Ending request: urn:request:e72c363f-e524-4eb0-a272-a30cac986318 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e72c363f-e524-4eb0-a272-a30cac986318', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:37,636 [243] [DEBUG] [app] Ending request: urn:request:9dc825f7-9aad-4276-a216-38294d0058de (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9dc825f7-9aad-4276-a216-38294d0058de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:37,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:37,647 [239] [DEBUG] [app] Ending request: urn:request:5c05a087-426a-4ba5-88bf-50d919e116bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5c05a087-426a-4ba5-88bf-50d919e116bf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:37,647 [241] [DEBUG] [app] Ending request: urn:request:b374975d-4138-49bc-9aaf-8e3408637812 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b374975d-4138-49bc-9aaf-8e3408637812', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:40:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:40:38,432 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:40:38,437 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:40:38,440 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:40:38,443 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:40:38,445 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:40:38,451 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:40:38,454 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:40:38,461 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:40:38,465 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:40:38,554 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:40:38,604 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:40:38,607 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:40:38,610 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:40:38,613 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:40:38,615 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:40:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:43.935629+00:00 (in 5.200115 seconds) notificationworker stdout | 2023-01-17 05:40:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:48 UTC)" (scheduled at 2023-01-17 05:40:38.735066+00:00) notificationworker stdout | 2023-01-17 05:40:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 38, 735892), True, datetime.datetime(2023, 1, 17, 5, 40, 38, 735892), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:38,747 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:38,747 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:38,748 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:40:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:40:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:48.738458+00:00 (in 9.707193 seconds) repositorygcworker stdout | 2023-01-17 05:40:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:39 UTC)" (scheduled at 2023-01-17 05:40:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:40:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:40:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 39, 31708), True, datetime.datetime(2023, 1, 17, 5, 40, 39, 31708), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:40:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:40:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:40:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:39,145 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:40:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:52.837365+00:00 (in 13.606895 seconds) securityworker stdout | 2023-01-17 05:40:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:09 UTC)" (scheduled at 2023-01-17 05:40:39.230038+00:00) securityworker stdout | 2023-01-17 05:40:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:40:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:40:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:40:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:40:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:40:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:40:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:40:39,930 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:40:39,955 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:40:39,965 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:40:39,969 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:40:41,998 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:42,007 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:42,007 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:42,013 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:48.735066+00:00 (in 4.798961 seconds) notificationworker stdout | 2023-01-17 05:40:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:43 UTC)" (scheduled at 2023-01-17 05:40:43.935629+00:00) notificationworker stdout | 2023-01-17 05:40:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:40:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:40:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:40:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:59.337413+00:00 (in 11.404442 seconds) namespacegcworker stdout | 2023-01-17 05:40:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:47 UTC)" (scheduled at 2023-01-17 05:40:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:40:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:40:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:58.735066+00:00 (in 9.999495 seconds) notificationworker stdout | 2023-01-17 05:40:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:58 UTC)" (scheduled at 2023-01-17 05:40:48.735066+00:00) notificationworker stdout | 2023-01-17 05:40:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 48, 735955), True, datetime.datetime(2023, 1, 17, 5, 40, 48, 735955), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:40:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:40:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:39.030814+00:00 (in 50.291896 seconds) repositorygcworker stdout | 2023-01-17 05:40:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:48 UTC)" (scheduled at 2023-01-17 05:40:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:40:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:40:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:40:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:40:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:24.641816+00:00 (in 35.811102 seconds) exportactionlogsworker stdout | 2023-01-17 05:40:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:48 UTC)" (scheduled at 2023-01-17 05:40:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:40:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:40:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 48, 831085), True, datetime.datetime(2023, 1, 17, 5, 40, 48, 831085), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:40:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:40:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:40:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:48 UTC)" executed successfully builder stdout | 2023-01-17 05:40:52,023 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:52,032 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:52,032 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:52,038 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:52,626 [241] [DEBUG] [app] Starting request: urn:request:6630d270-09d3-4a78-a621-d0ae14290054 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:52,626 [244] [DEBUG] [app] Starting request: urn:request:9d963ea9-1f5e-4e7b-8056-a2819b9095d3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:40:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:52,631 [242] [DEBUG] [app] Starting request: urn:request:5b04a438-2023-4e34-84c5-76d796324bd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:52,631 [240] [DEBUG] [app] Starting request: urn:request:1c0144b1-b49f-4b6c-a794-29a9af93eb59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:52,631 [242] [DEBUG] [app] Ending request: urn:request:5b04a438-2023-4e34-84c5-76d796324bd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5b04a438-2023-4e34-84c5-76d796324bd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:40:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:40:52,631 [240] [DEBUG] [app] Ending request: urn:request:1c0144b1-b49f-4b6c-a794-29a9af93eb59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c0144b1-b49f-4b6c-a794-29a9af93eb59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:40:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:52,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:52,637 [243] [DEBUG] [app] Starting request: urn:request:547f7896-de21-4d7f-be77-ab6dfb8f105c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:52,637 [244] [DEBUG] [app] Starting request: urn:request:68be698c-6423-4b13-bdf9-87fcf73ac0f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:52,637 [243] [DEBUG] [app] Ending request: urn:request:547f7896-de21-4d7f-be77-ab6dfb8f105c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:547f7896-de21-4d7f-be77-ab6dfb8f105c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:52,637 [244] [DEBUG] [app] Ending request: urn:request:68be698c-6423-4b13-bdf9-87fcf73ac0f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68be698c-6423-4b13-bdf9-87fcf73ac0f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:52,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:52,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:52,648 [244] [DEBUG] [app] Ending request: urn:request:9d963ea9-1f5e-4e7b-8056-a2819b9095d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d963ea9-1f5e-4e7b-8056-a2819b9095d3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:52,648 [241] [DEBUG] [app] Ending request: urn:request:6630d270-09d3-4a78-a621-d0ae14290054 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6630d270-09d3-4a78-a621-d0ae14290054', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:40:52,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:40:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:40:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:09.230038+00:00 (in 16.392216 seconds) securityworker stdout | 2023-01-17 05:40:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:22 UTC)" (scheduled at 2023-01-17 05:40:52.837365+00:00) securityworker stdout | 2023-01-17 05:40:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:40:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:40:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:27.226424+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2023-01-17 05:40:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:27 UTC)" (scheduled at 2023-01-17 05:40:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:40:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 40, 57, 227288), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:40:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:40:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:40:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:08.735066+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 05:40:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:08 UTC)" (scheduled at 2023-01-17 05:40:58.735066+00:00) notificationworker stdout | 2023-01-17 05:40:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 58, 735995), True, datetime.datetime(2023, 1, 17, 5, 40, 58, 735995), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:58,752 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:58,752 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:58,752 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:40:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:40:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:47.932490+00:00 (in 48.594594 seconds) namespacegcworker stdout | 2023-01-17 05:40:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:59 UTC)" (scheduled at 2023-01-17 05:40:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:40:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:40:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 59, 338304), True, datetime.datetime(2023, 1, 17, 5, 40, 59, 338304), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:40:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:40:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:40:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:41:00,976 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:41:01,169 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:41:01,404 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:41:02,038 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:02,048 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:02,048 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:02,054 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:41:02,192 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:41:02,264 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:41:03,516 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:41:04,225 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:41:04,362 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:41:04,699 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositoryactioncounter stdout | 2023-01-17 05:41:05,681 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:41:05,684 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gcworker stdout | 2023-01-17 05:41:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:41:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:35.848675+00:00 (in 29.999545 seconds) gcworker stdout | 2023-01-17 05:41:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:35 UTC)" (scheduled at 2023-01-17 05:41:05.848675+00:00) gcworker stdout | 2023-01-17 05:41:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724465849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:41:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:41:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:41:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:41:06,072 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:41:06,074 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:41:06,077 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:41:06,556 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:41:07,342 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:41:07,363 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:41:07,625 [239] [DEBUG] [app] Starting request: urn:request:c3e0fbdb-5c38-4f04-973a-aa8fcc56af7d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:07,625 [244] [DEBUG] [app] Starting request: urn:request:4e671f87-8892-47de-9490-2a3be8e88be6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:07,630 [242] [DEBUG] [app] Starting request: urn:request:8b32f9f4-5198-42c7-beba-fabcf431c630 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,631 [232] [DEBUG] [app] Starting request: urn:request:0aab6e5f-3551-4223-9ac6-b3c23f9a131d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,631 [242] [DEBUG] [app] Ending request: urn:request:8b32f9f4-5198-42c7-beba-fabcf431c630 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b32f9f4-5198-42c7-beba-fabcf431c630', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:41:07,631 [232] [DEBUG] [app] Ending request: urn:request:0aab6e5f-3551-4223-9ac6-b3c23f9a131d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0aab6e5f-3551-4223-9ac6-b3c23f9a131d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:41:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:41:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:07,636 [243] [DEBUG] [app] Starting request: urn:request:4a5acfdf-2fc5-4b1f-b058-3045240e2a79 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:07,636 [241] [DEBUG] [app] Starting request: urn:request:fe7c756b-8004-4121-a1d4-ad0d97a87af2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:07,636 [243] [DEBUG] [app] Ending request: urn:request:4a5acfdf-2fc5-4b1f-b058-3045240e2a79 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a5acfdf-2fc5-4b1f-b058-3045240e2a79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:07,636 [241] [DEBUG] [app] Ending request: urn:request:fe7c756b-8004-4121-a1d4-ad0d97a87af2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe7c756b-8004-4121-a1d4-ad0d97a87af2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:07,647 [244] [DEBUG] [app] Ending request: urn:request:4e671f87-8892-47de-9490-2a3be8e88be6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4e671f87-8892-47de-9490-2a3be8e88be6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:07,647 [239] [DEBUG] [app] Ending request: urn:request:c3e0fbdb-5c38-4f04-973a-aa8fcc56af7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3e0fbdb-5c38-4f04-973a-aa8fcc56af7d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:41:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:41:08,439 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:41:08,444 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:41:08,449 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:41:08,452 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:41:08,455 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:41:08,457 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:41:08,463 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:41:08,469 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:41:08,473 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:41:08,589 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:41:08,611 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:41:08,613 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:41:08,617 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:41:08,619 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:41:08,622 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:41:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:18.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:41:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:18 UTC)" (scheduled at 2023-01-17 05:41:08.735066+00:00) notificationworker stdout | 2023-01-17 05:41:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 8, 735935), True, datetime.datetime(2023, 1, 17, 5, 41, 8, 735935), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:09,181 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:41:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:22.837365+00:00 (in 13.606891 seconds) securityworker stdout | 2023-01-17 05:41:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:39 UTC)" (scheduled at 2023-01-17 05:41:09.230038+00:00) securityworker stdout | 2023-01-17 05:41:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:41:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:41:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:41:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:41:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:41:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:41:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:41:09,936 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:41:09,985 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:41:09,988 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:41:09,992 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:41:12,064 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:12,073 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:12,073 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:12,078 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:28.735066+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 05:41:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:28 UTC)" (scheduled at 2023-01-17 05:41:18.735066+00:00) notificationworker stdout | 2023-01-17 05:41:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 18, 735875), True, datetime.datetime(2023, 1, 17, 5, 41, 18, 735875), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:41:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:41:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:25.334138+00:00 (in 5.091433 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:41:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:20 UTC)" (scheduled at 2023-01-17 05:41:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:41:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:41:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 20, 243101), True, datetime.datetime(2023, 1, 17, 5, 41, 20, 243101), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:41:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:41:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:41:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:20 UTC)" executed successfully builder stdout | 2023-01-17 05:41:22,089 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:22,096 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:22,096 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:22,102 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:22,625 [241] [DEBUG] [app] Starting request: urn:request:f3221934-3a75-495b-b73a-731fb3a22a31 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:22,625 [239] [DEBUG] [app] Starting request: urn:request:2ed74d3e-b8b2-42ed-a38f-f0421173153d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:22,630 [240] [DEBUG] [app] Starting request: urn:request:0b18e062-a737-4c4d-ada5-19e8ce2ff03b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:22,630 [234] [DEBUG] [app] Starting request: urn:request:8e813604-d54c-457c-a3ad-4b26519bccb7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:22,631 [240] [DEBUG] [app] Ending request: urn:request:0b18e062-a737-4c4d-ada5-19e8ce2ff03b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b18e062-a737-4c4d-ada5-19e8ce2ff03b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:41:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:41:22,631 [234] [DEBUG] [app] Ending request: urn:request:8e813604-d54c-457c-a3ad-4b26519bccb7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8e813604-d54c-457c-a3ad-4b26519bccb7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:41:22,631 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:41:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:22,635 [244] [DEBUG] [app] Starting request: urn:request:18ea7967-3c0f-4e0c-8498-d3830a4b4ecd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:22,635 [241] [DEBUG] [app] Starting request: urn:request:d8923ef5-41d9-44a5-8a4a-c297354abec7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:22,636 [241] [DEBUG] [app] Ending request: urn:request:d8923ef5-41d9-44a5-8a4a-c297354abec7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8923ef5-41d9-44a5-8a4a-c297354abec7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:22,636 [244] [DEBUG] [app] Ending request: urn:request:18ea7967-3c0f-4e0c-8498-d3830a4b4ecd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18ea7967-3c0f-4e0c-8498-d3830a4b4ecd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:41:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:22,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:22,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:22,647 [241] [DEBUG] [app] Ending request: urn:request:f3221934-3a75-495b-b73a-731fb3a22a31 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3221934-3a75-495b-b73a-731fb3a22a31', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:22,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:22,647 [239] [DEBUG] [app] Ending request: urn:request:2ed74d3e-b8b2-42ed-a38f-f0421173153d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ed74d3e-b8b2-42ed-a38f-f0421173153d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:22,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:41:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:41:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:39.230038+00:00 (in 16.392210 seconds) securityworker stdout | 2023-01-17 05:41:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:52 UTC)" (scheduled at 2023-01-17 05:41:22.837365+00:00) securityworker stdout | 2023-01-17 05:41:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:22,858 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:22,859 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:22,859 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:41:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:41:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:44.641497+00:00 (in 19.999237 seconds) exportactionlogsworker stdout | 2023-01-17 05:41:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:24 UTC)" (scheduled at 2023-01-17 05:41:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:41:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:41:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:41:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:41:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:39.333860+00:00 (in 13.999228 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:41:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:25 UTC)" (scheduled at 2023-01-17 05:41:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:41:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:41:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:41:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:41:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:57.226424+00:00 (in 29.999559 seconds) buildlogsarchiver stdout | 2023-01-17 05:41:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:57 UTC)" (scheduled at 2023-01-17 05:41:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:41:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 41, 27, 227194), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:41:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:41:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:41:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:38.735066+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:41:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:38 UTC)" (scheduled at 2023-01-17 05:41:28.735066+00:00) notificationworker stdout | 2023-01-17 05:41:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 28, 735978), True, datetime.datetime(2023, 1, 17, 5, 41, 28, 735978), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:41:31,011 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:41:31,204 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:41:31,430 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:41:32,112 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:32,122 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:32,122 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:32,127 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:41:32,228 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:41:32,300 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:41:33,534 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:41:34,261 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:41:34,397 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:41:34,731 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositoryactioncounter stdout | 2023-01-17 05:41:35,716 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:41:35,718 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gcworker stdout | 2023-01-17 05:41:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:41:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:05.848675+00:00 (in 29.999553 seconds) gcworker stdout | 2023-01-17 05:41:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:05 UTC)" (scheduled at 2023-01-17 05:41:35.848675+00:00) gcworker stdout | 2023-01-17 05:41:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724495849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:41:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:41:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:41:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:41:36,079 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:41:36,081 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:41:36,084 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:41:36,579 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:41:37,377 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:41:37,398 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:41:37,626 [243] [DEBUG] [app] Starting request: urn:request:94198850-9c12-4629-a3ec-76162d93662f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:37,626 [244] [DEBUG] [app] Starting request: urn:request:ba0dc6f1-f3f1-4c30-931a-96ab0400f608 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:37,631 [242] [DEBUG] [app] Starting request: urn:request:1342ea3d-bacc-4514-95ec-80c6f29a3ca9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:37,631 [240] [DEBUG] [app] Starting request: urn:request:c63fe391-71e9-45d4-9440-228b8e991741 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:37,631 [242] [DEBUG] [app] Ending request: urn:request:1342ea3d-bacc-4514-95ec-80c6f29a3ca9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1342ea3d-bacc-4514-95ec-80c6f29a3ca9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:41:37,632 [240] [DEBUG] [app] Ending request: urn:request:c63fe391-71e9-45d4-9440-228b8e991741 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c63fe391-71e9-45d4-9440-228b8e991741', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:41:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:41:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:37,636 [241] [DEBUG] [app] Starting request: urn:request:0cbc036c-2568-4507-ab83-38d8f556a929 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:37,636 [244] [DEBUG] [app] Starting request: urn:request:4326a286-c42e-4fe6-b9d8-901174e4b021 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:37,636 [241] [DEBUG] [app] Ending request: urn:request:0cbc036c-2568-4507-ab83-38d8f556a929 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0cbc036c-2568-4507-ab83-38d8f556a929', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:37,636 [244] [DEBUG] [app] Ending request: urn:request:4326a286-c42e-4fe6-b9d8-901174e4b021 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4326a286-c42e-4fe6-b9d8-901174e4b021', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:37,647 [244] [DEBUG] [app] Ending request: urn:request:ba0dc6f1-f3f1-4c30-931a-96ab0400f608 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba0dc6f1-f3f1-4c30-931a-96ab0400f608', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:37,647 [243] [DEBUG] [app] Ending request: urn:request:94198850-9c12-4629-a3ec-76162d93662f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94198850-9c12-4629-a3ec-76162d93662f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:41:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:41:38,446 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:41:38,452 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:41:38,457 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:41:38,461 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:41:38,463 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:41:38,465 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:41:38,471 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:41:38,477 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:41:38,481 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:41:38,594 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:41:38,617 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:41:38,619 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:41:38,622 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:41:38,625 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:41:38,627 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:41:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:43.935629+00:00 (in 5.200109 seconds) notificationworker stdout | 2023-01-17 05:41:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:48 UTC)" (scheduled at 2023-01-17 05:41:38.735066+00:00) notificationworker stdout | 2023-01-17 05:41:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:38,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 38, 735978), True, datetime.datetime(2023, 1, 17, 5, 41, 38, 735978), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:41:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:41:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:48.738458+00:00 (in 9.707218 seconds) repositorygcworker stdout | 2023-01-17 05:41:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:39 UTC)" (scheduled at 2023-01-17 05:41:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:41:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:41:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 39, 31700), True, datetime.datetime(2023, 1, 17, 5, 41, 39, 31700), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:41:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:41:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:41:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:39,218 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:41:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:52.837365+00:00 (in 13.606896 seconds) securityworker stdout | 2023-01-17 05:41:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:09 UTC)" (scheduled at 2023-01-17 05:41:39.230038+00:00) securityworker stdout | 2023-01-17 05:41:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:41:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:41:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:41:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:41:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:41:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:41:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:41:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:41:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:20.242198+00:00 (in 40.907852 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:41:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:39 UTC)" (scheduled at 2023-01-17 05:41:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:41:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 41, 39, 334741), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:41:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 39, 334741), True, datetime.datetime(2023, 1, 17, 5, 41, 39, 334741), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:41:39,344 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 39, 334741), True, datetime.datetime(2023, 1, 17, 5, 41, 39, 334741), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 41, 39, 334741), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:41:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:41:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:41:39,972 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:41:40,020 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:41:40,023 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:41:40,026 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:41:42,128 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:42,136 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:42,137 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:42,142 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:48.735066+00:00 (in 4.798969 seconds) notificationworker stdout | 2023-01-17 05:41:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:43 UTC)" (scheduled at 2023-01-17 05:41:43.935629+00:00) notificationworker stdout | 2023-01-17 05:41:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:41:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:41:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:41:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:48.830274+00:00 (in 4.188315 seconds) exportactionlogsworker stdout | 2023-01-17 05:41:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:44 UTC)" (scheduled at 2023-01-17 05:41:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:41:44,643 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 41, 44, 642728), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:41:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 44, 642728), True, datetime.datetime(2023, 1, 17, 5, 41, 44, 642728), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:41:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 44, 642728), True, datetime.datetime(2023, 1, 17, 5, 41, 44, 642728), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 41, 44, 642728), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:41:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:41:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:41:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:41:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:59.337413+00:00 (in 11.404481 seconds) namespacegcworker stdout | 2023-01-17 05:41:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:47 UTC)" (scheduled at 2023-01-17 05:41:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:41:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:41:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:58.735066+00:00 (in 9.999480 seconds) notificationworker stdout | 2023-01-17 05:41:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:58 UTC)" (scheduled at 2023-01-17 05:41:48.735066+00:00) notificationworker stdout | 2023-01-17 05:41:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 48, 736004), True, datetime.datetime(2023, 1, 17, 5, 41, 48, 736004), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:41:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:41:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:04.738062+00:00 (in 15.999138 seconds) repositorygcworker stdout | 2023-01-17 05:41:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:48 UTC)" (scheduled at 2023-01-17 05:41:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:41:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:41:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:41:48,831 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:41:48,831 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:24.641816+00:00 (in 35.810691 seconds) exportactionlogsworker stdout | 2023-01-17 05:41:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:48 UTC)" (scheduled at 2023-01-17 05:41:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:41:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:41:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 48, 831524), True, datetime.datetime(2023, 1, 17, 5, 41, 48, 831524), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:41:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:41:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:41:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:48 UTC)" executed successfully builder stdout | 2023-01-17 05:41:52,153 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:52,162 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:52,162 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:52,167 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:52,625 [243] [DEBUG] [app] Starting request: urn:request:b16e8df6-cfe6-4f74-9d71-c8c9b793d448 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:52,626 [244] [DEBUG] [app] Starting request: urn:request:00d9ee89-794c-402d-b44a-eb2cfc2a000a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:41:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:52,631 [232] [DEBUG] [app] Starting request: urn:request:613e0de1-3e0e-4d61-9124-ea82ce2e9991 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:52,631 [240] [DEBUG] [app] Starting request: urn:request:d5dc849f-770b-44a0-84a4-4d5e74a053a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:52,631 [232] [DEBUG] [app] Ending request: urn:request:613e0de1-3e0e-4d61-9124-ea82ce2e9991 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:613e0de1-3e0e-4d61-9124-ea82ce2e9991', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:41:52,631 [240] [DEBUG] [app] Ending request: urn:request:d5dc849f-770b-44a0-84a4-4d5e74a053a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d5dc849f-770b-44a0-84a4-4d5e74a053a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:41:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:41:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:41:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:52,636 [241] [DEBUG] [app] Starting request: urn:request:a5c98ff7-8677-4c85-910a-240afecb11fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:52,636 [244] [DEBUG] [app] Starting request: urn:request:e70ac9a5-74aa-4879-b974-86ce24a8dad7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:52,636 [241] [DEBUG] [app] Ending request: urn:request:a5c98ff7-8677-4c85-910a-240afecb11fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a5c98ff7-8677-4c85-910a-240afecb11fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:52,636 [244] [DEBUG] [app] Ending request: urn:request:e70ac9a5-74aa-4879-b974-86ce24a8dad7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e70ac9a5-74aa-4879-b974-86ce24a8dad7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:52,647 [244] [DEBUG] [app] Ending request: urn:request:00d9ee89-794c-402d-b44a-eb2cfc2a000a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:00d9ee89-794c-402d-b44a-eb2cfc2a000a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:52,647 [243] [DEBUG] [app] Ending request: urn:request:b16e8df6-cfe6-4f74-9d71-c8c9b793d448 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b16e8df6-cfe6-4f74-9d71-c8c9b793d448', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:41:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:41:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:09.230038+00:00 (in 16.392230 seconds) securityworker stdout | 2023-01-17 05:41:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:22 UTC)" (scheduled at 2023-01-17 05:41:52.837365+00:00) securityworker stdout | 2023-01-17 05:41:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:41:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:41:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:27.226424+00:00 (in 29.999576 seconds) buildlogsarchiver stdout | 2023-01-17 05:41:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:27 UTC)" (scheduled at 2023-01-17 05:41:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:41:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 41, 57, 227197), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:41:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:41:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:41:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:08.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 05:41:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:08 UTC)" (scheduled at 2023-01-17 05:41:58.735066+00:00) notificationworker stdout | 2023-01-17 05:41:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 58, 735986), True, datetime.datetime(2023, 1, 17, 5, 41, 58, 735986), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:41:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:41:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:47.932490+00:00 (in 48.594584 seconds) namespacegcworker stdout | 2023-01-17 05:41:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:59 UTC)" (scheduled at 2023-01-17 05:41:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:41:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:41:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 59, 338560), True, datetime.datetime(2023, 1, 17, 5, 41, 59, 338560), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:41:59,347 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:41:59,347 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:41:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:01,047 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:42:01,210 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:42:01,444 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:42:02,177 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:02,186 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:02,186 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:02,193 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:42:02,263 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:42:02,317 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:42:03,570 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:42:04,279 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:42:04,433 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} repositorygcworker stdout | 2023-01-17 05:42:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:42:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:39.030814+00:00 (in 34.292294 seconds) repositorygcworker stdout | 2023-01-17 05:42:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:47:04 UTC)" (scheduled at 2023-01-17 05:42:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:42:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 42, 4, 738871), 'repositorygc/%']) namespacegcworker stdout | 2023-01-17 05:42:04,740 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositorygcworker stdout | 2023-01-17 05:42:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 42, 4, 738871), True, datetime.datetime(2023, 1, 17, 5, 42, 4, 738871), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:42:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 42, 4, 738871), True, datetime.datetime(2023, 1, 17, 5, 42, 4, 738871), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 42, 4, 738871), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:42:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:42:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:47:04 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:42:05,752 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:42:05,755 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gcworker stdout | 2023-01-17 05:42:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:42:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:35.848675+00:00 (in 29.999565 seconds) gcworker stdout | 2023-01-17 05:42:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:35 UTC)" (scheduled at 2023-01-17 05:42:05.848675+00:00) gcworker stdout | 2023-01-17 05:42:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724525849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:42:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:42:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:42:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:42:06,086 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:42:06,089 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:42:06,091 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:42:06,597 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:42:07,413 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:42:07,415 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:42:07,625 [239] [DEBUG] [app] Starting request: urn:request:ffb4153c-d8ba-43ad-bec3-d8f59633b94f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:07,625 [244] [DEBUG] [app] Starting request: urn:request:c533ffca-fec8-4c7a-8e6a-b07091e5801f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:07,630 [240] [DEBUG] [app] Starting request: urn:request:b636ec39-1adc-43a9-941a-050390488af2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,630 [236] [DEBUG] [app] Starting request: urn:request:9a8d852d-6106-4db5-a7b7-e38171aedbc0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,631 [240] [DEBUG] [app] Ending request: urn:request:b636ec39-1adc-43a9-941a-050390488af2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b636ec39-1adc-43a9-941a-050390488af2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:42:07,631 [236] [DEBUG] [app] Ending request: urn:request:9a8d852d-6106-4db5-a7b7-e38171aedbc0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a8d852d-6106-4db5-a7b7-e38171aedbc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:42:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:42:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:07,635 [244] [DEBUG] [app] Starting request: urn:request:283dfc72-244c-4104-ba99-7ce8051c5b44 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:07,635 [244] [DEBUG] [app] Ending request: urn:request:283dfc72-244c-4104-ba99-7ce8051c5b44 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:283dfc72-244c-4104-ba99-7ce8051c5b44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:07,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:07,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:07,637 [241] [DEBUG] [app] Starting request: urn:request:93744fec-a29e-4622-9b67-a484dfc746bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:07,637 [241] [DEBUG] [app] Ending request: urn:request:93744fec-a29e-4622-9b67-a484dfc746bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93744fec-a29e-4622-9b67-a484dfc746bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:07,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:07,646 [239] [DEBUG] [app] Ending request: urn:request:ffb4153c-d8ba-43ad-bec3-d8f59633b94f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ffb4153c-d8ba-43ad-bec3-d8f59633b94f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:07,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:42:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:42:07,648 [244] [DEBUG] [app] Ending request: urn:request:c533ffca-fec8-4c7a-8e6a-b07091e5801f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c533ffca-fec8-4c7a-8e6a-b07091e5801f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-registry stdout | 2023-01-17 05:42:08,459 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:42:08,461 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:42:08,464 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:42:08,469 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:42:08,473 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:42:08,475 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:42:08,479 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:42:08,484 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:42:08,489 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} storagereplication stdout | 2023-01-17 05:42:08,620 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 05:42:08,624 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:42:08,626 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:42:08,629 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:42:08,631 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:42:08,634 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:42:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:18.735066+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 05:42:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:18 UTC)" (scheduled at 2023-01-17 05:42:08.735066+00:00) notificationworker stdout | 2023-01-17 05:42:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 8, 735938), True, datetime.datetime(2023, 1, 17, 5, 42, 8, 735938), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:18 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:42:09,227 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:42:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:22.837365+00:00 (in 13.606887 seconds) securityworker stdout | 2023-01-17 05:42:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:39 UTC)" (scheduled at 2023-01-17 05:42:09.230038+00:00) securityworker stdout | 2023-01-17 05:42:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:42:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:42:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:42:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:42:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:42:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:42:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:39 UTC)" executed successfully gcworker stdout | 2023-01-17 05:42:09,984 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 05:42:10,046 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} blobuploadcleanupworker stdout | 2023-01-17 05:42:10,048 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:42:10,057 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} builder stdout | 2023-01-17 05:42:12,203 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:12,212 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:12,212 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:12,218 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:28.735066+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:42:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:28 UTC)" (scheduled at 2023-01-17 05:42:18.735066+00:00) notificationworker stdout | 2023-01-17 05:42:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 18, 735942), True, datetime.datetime(2023, 1, 17, 5, 42, 18, 735942), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:42:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:42:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:25.334138+00:00 (in 5.091468 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:42:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:20 UTC)" (scheduled at 2023-01-17 05:42:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:42:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:42:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 20, 243103), True, datetime.datetime(2023, 1, 17, 5, 42, 20, 243103), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:42:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:42:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:42:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:20 UTC)" executed successfully builder stdout | 2023-01-17 05:42:22,228 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:22,236 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:22,236 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:22,242 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:22,625 [244] [DEBUG] [app] Starting request: urn:request:428c3e4c-98ac-4067-9d46-d3ae9403da76 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:22,625 [241] [DEBUG] [app] Starting request: urn:request:84703632-f690-4590-9cf5-ed68e7877c1a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:22,630 [235] [DEBUG] [app] Starting request: urn:request:3b751ce8-d322-4456-9f3e-2ec242ce069b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:22,630 [237] [DEBUG] [app] Starting request: urn:request:d2cced1f-fec6-47b6-a630-ea8240a1670e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:22,631 [235] [DEBUG] [app] Ending request: urn:request:3b751ce8-d322-4456-9f3e-2ec242ce069b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b751ce8-d322-4456-9f3e-2ec242ce069b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:42:22,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:42:22,631 [237] [DEBUG] [app] Ending request: urn:request:d2cced1f-fec6-47b6-a630-ea8240a1670e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2cced1f-fec6-47b6-a630-ea8240a1670e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:42:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:22,635 [243] [DEBUG] [app] Starting request: urn:request:a3893368-1f69-4922-aeef-223edc30033e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:22,636 [239] [DEBUG] [app] Starting request: urn:request:52bf34a8-bbdc-4e0e-8782-3343932ae901 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:22,636 [243] [DEBUG] [app] Ending request: urn:request:a3893368-1f69-4922-aeef-223edc30033e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3893368-1f69-4922-aeef-223edc30033e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:22,636 [239] [DEBUG] [app] Ending request: urn:request:52bf34a8-bbdc-4e0e-8782-3343932ae901 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:52bf34a8-bbdc-4e0e-8782-3343932ae901', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:42:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:22,647 [244] [DEBUG] [app] Ending request: urn:request:428c3e4c-98ac-4067-9d46-d3ae9403da76 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:428c3e4c-98ac-4067-9d46-d3ae9403da76', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:22,647 [241] [DEBUG] [app] Ending request: urn:request:84703632-f690-4590-9cf5-ed68e7877c1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84703632-f690-4590-9cf5-ed68e7877c1a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:42:22,647 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:42:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:42:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:39.230038+00:00 (in 16.392229 seconds) securityworker stdout | 2023-01-17 05:42:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:52 UTC)" (scheduled at 2023-01-17 05:42:22.837365+00:00) securityworker stdout | 2023-01-17 05:42:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:42:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:42:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:48.830274+00:00 (in 24.187984 seconds) exportactionlogsworker stdout | 2023-01-17 05:42:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:24 UTC)" (scheduled at 2023-01-17 05:42:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:42:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:42:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:42:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:42:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:20.242198+00:00 (in 54.907546 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:42:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:25 UTC)" (scheduled at 2023-01-17 05:42:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:42:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:42:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:42:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:42:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:57.226424+00:00 (in 29.999558 seconds) buildlogsarchiver stdout | 2023-01-17 05:42:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:57 UTC)" (scheduled at 2023-01-17 05:42:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:42:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 42, 27, 227254), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:42:27,237 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:42:27,237 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:42:27,237 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:42:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:38.735066+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:42:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:38 UTC)" (scheduled at 2023-01-17 05:42:28.735066+00:00) notificationworker stdout | 2023-01-17 05:42:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 28, 735976), True, datetime.datetime(2023, 1, 17, 5, 42, 28, 735976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:31,055 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:42:31,249 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:42:31,450 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 05:42:32,252 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:32,261 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:32,261 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:32,267 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:42:32,296 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:42:32,342 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:42:33,592 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:42:34,293 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:42:34,469 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:42:34,770 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositoryactioncounter stdout | 2023-01-17 05:42:35,760 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:42:35,763 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gcworker stdout | 2023-01-17 05:42:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:42:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:05.848675+00:00 (in 29.999505 seconds) gcworker stdout | 2023-01-17 05:42:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:05 UTC)" (scheduled at 2023-01-17 05:42:35.848675+00:00) gcworker stdout | 2023-01-17 05:42:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724555849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:42:35,863 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:42:35,863 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:42:35,863 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:42:36,094 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:42:36,097 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:42:36,099 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:42:36,632 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} queuecleanupworker stdout | 2023-01-17 05:42:37,449 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:42:37,452 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:42:37,625 [243] [DEBUG] [app] Starting request: urn:request:6882d1b8-8092-4389-831f-18bd71adc10a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:37,626 [244] [DEBUG] [app] Starting request: urn:request:93d77d4a-b838-4dea-a56e-f1c90dee3e2d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:37,631 [232] [DEBUG] [app] Starting request: urn:request:79b5ec90-b5a3-4573-b99e-c2d970349a6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:37,631 [242] [DEBUG] [app] Starting request: urn:request:b14640af-6e07-4dbc-a350-34c1138b0a59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:37,631 [232] [DEBUG] [app] Ending request: urn:request:79b5ec90-b5a3-4573-b99e-c2d970349a6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79b5ec90-b5a3-4573-b99e-c2d970349a6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:37,631 [242] [DEBUG] [app] Ending request: urn:request:b14640af-6e07-4dbc-a350-34c1138b0a59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b14640af-6e07-4dbc-a350-34c1138b0a59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:42:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:42:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:37,636 [239] [DEBUG] [app] Starting request: urn:request:efc973bc-79bc-459b-bd91-94ddad95be53 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:37,636 [244] [DEBUG] [app] Starting request: urn:request:36b9f660-385f-404f-a975-7ab19b246c96 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:37,636 [239] [DEBUG] [app] Ending request: urn:request:efc973bc-79bc-459b-bd91-94ddad95be53 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:efc973bc-79bc-459b-bd91-94ddad95be53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:37,636 [244] [DEBUG] [app] Ending request: urn:request:36b9f660-385f-404f-a975-7ab19b246c96 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36b9f660-385f-404f-a975-7ab19b246c96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:37,648 [244] [DEBUG] [app] Ending request: urn:request:93d77d4a-b838-4dea-a56e-f1c90dee3e2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93d77d4a-b838-4dea-a56e-f1c90dee3e2d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:37,648 [243] [DEBUG] [app] Ending request: urn:request:6882d1b8-8092-4389-831f-18bd71adc10a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6882d1b8-8092-4389-831f-18bd71adc10a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:42:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-registry stdout | 2023-01-17 05:42:38,470 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:42:38,473 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:42:38,475 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:42:38,477 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:42:38,482 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:42:38,485 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:42:38,488 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:42:38,491 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:42:38,496 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:42:38,631 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:42:38,633 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:42:38,636 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:42:38,639 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:42:38,641 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:42:38,654 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:42:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:43.935629+00:00 (in 5.200118 seconds) notificationworker stdout | 2023-01-17 05:42:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:48 UTC)" (scheduled at 2023-01-17 05:42:38.735066+00:00) notificationworker stdout | 2023-01-17 05:42:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 38, 735880), True, datetime.datetime(2023, 1, 17, 5, 42, 38, 735880), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:42:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:42:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:48.738458+00:00 (in 9.707225 seconds) repositorygcworker stdout | 2023-01-17 05:42:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:39 UTC)" (scheduled at 2023-01-17 05:42:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:42:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:42:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 39, 31641), True, datetime.datetime(2023, 1, 17, 5, 42, 39, 31641), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:42:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:42:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:42:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:52.837365+00:00 (in 13.606861 seconds) securityworker stdout | 2023-01-17 05:42:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:09 UTC)" (scheduled at 2023-01-17 05:42:39.230038+00:00) securityworker stdout | 2023-01-17 05:42:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:42:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:42:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:39,232 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:39,232 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state notificationworker stdout | 2023-01-17 05:42:39,234 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} securityworker stdout | 2023-01-17 05:42:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:39,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:39,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:42:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:09 UTC)" executed successfully gcworker stdout | 2023-01-17 05:42:39,990 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:42:40,056 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:42:40,074 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} expiredappspecifictokenworker stdout | 2023-01-17 05:42:40,077 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} builder stdout | 2023-01-17 05:42:42,277 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:42,286 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:42,286 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:42,292 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:48.735066+00:00 (in 4.798949 seconds) notificationworker stdout | 2023-01-17 05:42:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:43 UTC)" (scheduled at 2023-01-17 05:42:43.935629+00:00) notificationworker stdout | 2023-01-17 05:42:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:42:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:42:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:42:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:59.337413+00:00 (in 11.404422 seconds) namespacegcworker stdout | 2023-01-17 05:42:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:47 UTC)" (scheduled at 2023-01-17 05:42:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:42:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:42:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:42:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:58.735066+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:42:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:58 UTC)" (scheduled at 2023-01-17 05:42:48.735066+00:00) notificationworker stdout | 2023-01-17 05:42:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 48, 735958), True, datetime.datetime(2023, 1, 17, 5, 42, 48, 735958), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:42:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:42:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:39.030814+00:00 (in 50.291901 seconds) repositorygcworker stdout | 2023-01-17 05:42:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:48 UTC)" (scheduled at 2023-01-17 05:42:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:42:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:42:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:42:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:42:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:42:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:24.641816+00:00 (in 35.811086 seconds) exportactionlogsworker stdout | 2023-01-17 05:42:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:48 UTC)" (scheduled at 2023-01-17 05:42:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:42:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:42:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 48, 831081), True, datetime.datetime(2023, 1, 17, 5, 42, 48, 831081), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:42:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:42:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:42:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:48 UTC)" executed successfully builder stdout | 2023-01-17 05:42:52,302 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:52,311 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:52,311 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:52,317 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:52,626 [244] [DEBUG] [app] Starting request: urn:request:689dcb8a-e321-4106-a03b-26048018b89b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:52,626 [239] [DEBUG] [app] Starting request: urn:request:f9e7bea2-75af-4a62-9e14-7cddd07d86e6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:42:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:52,631 [240] [DEBUG] [app] Starting request: urn:request:9d9739cc-3e88-49f4-8f8d-aef479e7ddc5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,631 [232] [DEBUG] [app] Starting request: urn:request:a51d52de-2db7-4606-8881-8898f14c1026 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,632 [240] [DEBUG] [app] Ending request: urn:request:9d9739cc-3e88-49f4-8f8d-aef479e7ddc5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d9739cc-3e88-49f4-8f8d-aef479e7ddc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,632 [232] [DEBUG] [app] Ending request: urn:request:a51d52de-2db7-4606-8881-8898f14c1026 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a51d52de-2db7-4606-8881-8898f14c1026', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:42:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:42:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:52,636 [241] [DEBUG] [app] Starting request: urn:request:8d8c3f15-fe90-4981-a2dc-488d0969cda7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:52,637 [243] [DEBUG] [app] Starting request: urn:request:b6005157-7f16-45a7-be2b-cbe868312371 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:52,637 [241] [DEBUG] [app] Ending request: urn:request:8d8c3f15-fe90-4981-a2dc-488d0969cda7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d8c3f15-fe90-4981-a2dc-488d0969cda7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:52,637 [243] [DEBUG] [app] Ending request: urn:request:b6005157-7f16-45a7-be2b-cbe868312371 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6005157-7f16-45a7-be2b-cbe868312371', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:42:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:52,648 [244] [DEBUG] [app] Ending request: urn:request:689dcb8a-e321-4106-a03b-26048018b89b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:689dcb8a-e321-4106-a03b-26048018b89b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:52,648 [239] [DEBUG] [app] Ending request: urn:request:f9e7bea2-75af-4a62-9e14-7cddd07d86e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f9e7bea2-75af-4a62-9e14-7cddd07d86e6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:42:52,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:42:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:42:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:09.230038+00:00 (in 16.392203 seconds) securityworker stdout | 2023-01-17 05:42:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:22 UTC)" (scheduled at 2023-01-17 05:42:52.837365+00:00) securityworker stdout | 2023-01-17 05:42:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:42:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:42:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:27.226424+00:00 (in 29.999535 seconds) buildlogsarchiver stdout | 2023-01-17 05:42:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:27 UTC)" (scheduled at 2023-01-17 05:42:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:42:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 42, 57, 227263), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:42:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:42:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:42:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:42:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:08.735066+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:42:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:08 UTC)" (scheduled at 2023-01-17 05:42:58.735066+00:00) notificationworker stdout | 2023-01-17 05:42:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 58, 735899), True, datetime.datetime(2023, 1, 17, 5, 42, 58, 735899), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:42:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:42:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:47.932490+00:00 (in 48.594596 seconds) namespacegcworker stdout | 2023-01-17 05:42:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:59 UTC)" (scheduled at 2023-01-17 05:42:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:42:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:42:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 59, 338305), True, datetime.datetime(2023, 1, 17, 5, 42, 59, 338305), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:42:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:42:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:42:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:01,080 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:43:01,255 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:43:01,486 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:43:02,319 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 05:43:02,327 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:02,336 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:02,336 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:02,342 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:43:02,377 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:43:03,610 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:43:04,331 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:43:04,478 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:43:04,806 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} repositoryactioncounter stdout | 2023-01-17 05:43:05,777 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:43:05,780 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gcworker stdout | 2023-01-17 05:43:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:43:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:35.848675+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 05:43:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:35 UTC)" (scheduled at 2023-01-17 05:43:05.848675+00:00) gcworker stdout | 2023-01-17 05:43:05,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:43:05,856 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724585855, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:43:05,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:43:05,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:43:05,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:43:06,100 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:43:06,103 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:43:06,105 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:43:06,645 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:43:07,461 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:43:07,463 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:43:07,625 [241] [DEBUG] [app] Starting request: urn:request:a4496e95-7952-4b53-895d-9a2bc65e4017 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:07,626 [243] [DEBUG] [app] Starting request: urn:request:225a3fb0-111d-4d74-9544-de0dc0fda092 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:07,631 [242] [DEBUG] [app] Starting request: urn:request:7e16dbc4-9e02-4b3e-8764-a323bb312ea9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,631 [232] [DEBUG] [app] Starting request: urn:request:1b956856-b5fb-4ce2-9853-e9524827c30b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,631 [242] [DEBUG] [app] Ending request: urn:request:7e16dbc4-9e02-4b3e-8764-a323bb312ea9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7e16dbc4-9e02-4b3e-8764-a323bb312ea9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:43:07,631 [232] [DEBUG] [app] Ending request: urn:request:1b956856-b5fb-4ce2-9853-e9524827c30b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b956856-b5fb-4ce2-9853-e9524827c30b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:43:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:43:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:07,636 [239] [DEBUG] [app] Starting request: urn:request:fcaf3518-37d2-4f83-b608-b42cdcb02f20 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:07,636 [239] [DEBUG] [app] Ending request: urn:request:fcaf3518-37d2-4f83-b608-b42cdcb02f20 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fcaf3518-37d2-4f83-b608-b42cdcb02f20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:07,636 [244] [DEBUG] [app] Starting request: urn:request:293370b9-e8a9-4276-9b8e-d473ee2b1733 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:43:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:07,637 [244] [DEBUG] [app] Ending request: urn:request:293370b9-e8a9-4276-9b8e-d473ee2b1733 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:293370b9-e8a9-4276-9b8e-d473ee2b1733', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:43:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:07,647 [243] [DEBUG] [app] Ending request: urn:request:225a3fb0-111d-4d74-9544-de0dc0fda092 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:225a3fb0-111d-4d74-9544-de0dc0fda092', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:43:07,648 [241] [DEBUG] [app] Ending request: urn:request:a4496e95-7952-4b53-895d-9a2bc65e4017 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a4496e95-7952-4b53-895d-9a2bc65e4017', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:43:07,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:43:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:43:08,480 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:43:08,482 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:43:08,484 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:43:08,487 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:43:08,489 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:43:08,494 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:43:08,497 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:43:08,499 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:43:08,503 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:43:08,638 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:43:08,641 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:43:08,643 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:43:08,646 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:43:08,648 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:43:08,666 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:43:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:18.735066+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 05:43:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:18 UTC)" (scheduled at 2023-01-17 05:43:08.735066+00:00) notificationworker stdout | 2023-01-17 05:43:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 8, 735873), True, datetime.datetime(2023, 1, 17, 5, 43, 8, 735873), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:22.837365+00:00 (in 13.606874 seconds) securityworker stdout | 2023-01-17 05:43:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:39 UTC)" (scheduled at 2023-01-17 05:43:09.230038+00:00) securityworker stdout | 2023-01-17 05:43:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:43:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:43:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:43:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:43:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:43:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:43:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:09,249 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:43:10,027 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:43:10,091 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:43:10,094 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:43:10,100 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:43:12,349 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:12,358 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:12,358 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:12,364 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:28.735066+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:43:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:28 UTC)" (scheduled at 2023-01-17 05:43:18.735066+00:00) notificationworker stdout | 2023-01-17 05:43:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 18, 735991), True, datetime.datetime(2023, 1, 17, 5, 43, 18, 735991), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:43:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:43:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:25.334138+00:00 (in 5.091443 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:43:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:20 UTC)" (scheduled at 2023-01-17 05:43:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:43:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:43:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 20, 243120), True, datetime.datetime(2023, 1, 17, 5, 43, 20, 243120), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:43:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:43:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:43:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:20 UTC)" executed successfully builder stdout | 2023-01-17 05:43:22,374 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:22,382 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:22,383 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:22,388 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:22,625 [239] [DEBUG] [app] Starting request: urn:request:30741a22-a2c9-47e8-9c19-b168a2496b84 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:22,626 [243] [DEBUG] [app] Starting request: urn:request:3f627549-d905-406b-9a64-3ac7cc70fa20 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:22,631 [232] [DEBUG] [app] Starting request: urn:request:a33df371-25e7-4eb5-8476-55026c8a2e42 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:22,631 [235] [DEBUG] [app] Starting request: urn:request:a9e384a4-402c-4d2a-bdbf-dc6901c15d76 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:22,631 [232] [DEBUG] [app] Ending request: urn:request:a33df371-25e7-4eb5-8476-55026c8a2e42 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a33df371-25e7-4eb5-8476-55026c8a2e42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:22,631 [235] [DEBUG] [app] Ending request: urn:request:a9e384a4-402c-4d2a-bdbf-dc6901c15d76 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9e384a4-402c-4d2a-bdbf-dc6901c15d76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:43:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:43:22,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:22,637 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:22,637 [241] [DEBUG] [app] Starting request: urn:request:1edf2633-d5f3-432f-99eb-ac823ef191d6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:22,637 [241] [DEBUG] [app] Ending request: urn:request:1edf2633-d5f3-432f-99eb-ac823ef191d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1edf2633-d5f3-432f-99eb-ac823ef191d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:22,638 [244] [DEBUG] [app] Starting request: urn:request:8acdd39a-32ff-4d17-a436-196864b0cd39 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:22,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:22,638 [244] [DEBUG] [app] Ending request: urn:request:8acdd39a-32ff-4d17-a436-196864b0cd39 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8acdd39a-32ff-4d17-a436-196864b0cd39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:22,639 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:22,639 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:43:22,639 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:22,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:22,649 [243] [DEBUG] [app] Ending request: urn:request:3f627549-d905-406b-9a64-3ac7cc70fa20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f627549-d905-406b-9a64-3ac7cc70fa20', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:22,649 [239] [DEBUG] [app] Ending request: urn:request:30741a22-a2c9-47e8-9c19-b168a2496b84 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:30741a22-a2c9-47e8-9c19-b168a2496b84', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:22,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:22,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:43:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) securityworker stdout | 2023-01-17 05:43:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:39.230038+00:00 (in 16.392222 seconds) securityworker stdout | 2023-01-17 05:43:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:52 UTC)" (scheduled at 2023-01-17 05:43:22.837365+00:00) securityworker stdout | 2023-01-17 05:43:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:43:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:43:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:48.830274+00:00 (in 24.188016 seconds) exportactionlogsworker stdout | 2023-01-17 05:43:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:24 UTC)" (scheduled at 2023-01-17 05:43:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:43:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:43:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:43:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:43:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:20.242198+00:00 (in 54.907557 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:43:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:25 UTC)" (scheduled at 2023-01-17 05:43:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:43:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:43:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:43:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:43:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:57.226424+00:00 (in 29.999541 seconds) buildlogsarchiver stdout | 2023-01-17 05:43:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:57 UTC)" (scheduled at 2023-01-17 05:43:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:43:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 43, 27, 227243), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:43:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:43:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:43:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:38.735066+00:00 (in 9.999472 seconds) notificationworker stdout | 2023-01-17 05:43:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:38 UTC)" (scheduled at 2023-01-17 05:43:28.735066+00:00) notificationworker stdout | 2023-01-17 05:43:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 28, 736028), True, datetime.datetime(2023, 1, 17, 5, 43, 28, 736028), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:31,092 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:43:31,261 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:43:31,520 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:43:32,331 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 05:43:32,399 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:32,408 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:32,408 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics buildlogsarchiver stdout | 2023-01-17 05:43:32,412 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:43:32,414 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:43:33,646 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:43:34,348 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:43:34,509 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:43:34,838 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:43:35,786 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:43:35,789 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:43:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:43:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:05.848675+00:00 (in 29.999478 seconds) gcworker stdout | 2023-01-17 05:43:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:05 UTC)" (scheduled at 2023-01-17 05:43:35.848675+00:00) gcworker stdout | 2023-01-17 05:43:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724615849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:43:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:43:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:43:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:43:36,108 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:43:36,111 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:43:36,113 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:43:36,680 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:43:37,470 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:43:37,492 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:43:37,625 [244] [DEBUG] [app] Starting request: urn:request:fe5ff03a-99d5-4020-953f-3d09da6dad25 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:37,625 [243] [DEBUG] [app] Starting request: urn:request:75c260f3-30cd-41e4-8e8f-7e0c5bde759e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:37,630 [242] [DEBUG] [app] Starting request: urn:request:1e9e4300-c6a0-4d0a-8cc9-5048a602228f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:37,630 [237] [DEBUG] [app] Starting request: urn:request:f8bf6cfd-8250-4a40-be10-4dc3bc8de882 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:37,630 [242] [DEBUG] [app] Ending request: urn:request:1e9e4300-c6a0-4d0a-8cc9-5048a602228f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e9e4300-c6a0-4d0a-8cc9-5048a602228f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:43:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:43:37,631 [237] [DEBUG] [app] Ending request: urn:request:f8bf6cfd-8250-4a40-be10-4dc3bc8de882 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f8bf6cfd-8250-4a40-be10-4dc3bc8de882', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:43:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:37,635 [241] [DEBUG] [app] Starting request: urn:request:7092fc83-01e9-48a7-b42b-821879ee1e83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:37,635 [241] [DEBUG] [app] Ending request: urn:request:7092fc83-01e9-48a7-b42b-821879ee1e83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7092fc83-01e9-48a7-b42b-821879ee1e83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:37,635 [239] [DEBUG] [app] Starting request: urn:request:b845db73-38d1-4540-a3e3-2e06516b9c58 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:37,636 [239] [DEBUG] [app] Ending request: urn:request:b845db73-38d1-4540-a3e3-2e06516b9c58 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b845db73-38d1-4540-a3e3-2e06516b9c58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:43:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:37,646 [244] [DEBUG] [app] Ending request: urn:request:fe5ff03a-99d5-4020-953f-3d09da6dad25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fe5ff03a-99d5-4020-953f-3d09da6dad25', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:37,647 [243] [DEBUG] [app] Ending request: urn:request:75c260f3-30cd-41e4-8e8f-7e0c5bde759e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75c260f3-30cd-41e4-8e8f-7e0c5bde759e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:37,647 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:43:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:43:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:43:38,489 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:43:38,493 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:43:38,495 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:43:38,497 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:43:38,499 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:43:38,502 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:43:38,505 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:43:38,507 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 05:43:38,510 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 05:43:38,645 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:43:38,647 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:43:38,650 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:43:38,652 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:43:38,654 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:43:38,701 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:43:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:43.935629+00:00 (in 5.200111 seconds) notificationworker stdout | 2023-01-17 05:43:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:48 UTC)" (scheduled at 2023-01-17 05:43:38.735066+00:00) notificationworker stdout | 2023-01-17 05:43:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 38, 735874), True, datetime.datetime(2023, 1, 17, 5, 43, 38, 735874), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:43:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:43:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:48.738458+00:00 (in 9.707186 seconds) repositorygcworker stdout | 2023-01-17 05:43:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:39 UTC)" (scheduled at 2023-01-17 05:43:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:43:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:43:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 39, 31656), True, datetime.datetime(2023, 1, 17, 5, 43, 39, 31656), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:43:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:43:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:43:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:52.837365+00:00 (in 13.606875 seconds) securityworker stdout | 2023-01-17 05:43:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:09 UTC)" (scheduled at 2023-01-17 05:43:39.230038+00:00) securityworker stdout | 2023-01-17 05:43:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:43:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:43:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:43:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:39,256 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:43:40,048 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:43:40,098 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:43:40,126 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:43:40,135 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:43:42,424 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:42,434 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:42,434 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:42,439 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:48.735066+00:00 (in 4.798987 seconds) notificationworker stdout | 2023-01-17 05:43:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:43 UTC)" (scheduled at 2023-01-17 05:43:43.935629+00:00) notificationworker stdout | 2023-01-17 05:43:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:43:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:43:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:43:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:59.337413+00:00 (in 11.404459 seconds) namespacegcworker stdout | 2023-01-17 05:43:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:47 UTC)" (scheduled at 2023-01-17 05:43:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:43:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:43:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:56.935055+00:00 (in 8.199506 seconds) notificationworker stdout | 2023-01-17 05:43:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:58 UTC)" (scheduled at 2023-01-17 05:43:48.735066+00:00) notificationworker stdout | 2023-01-17 05:43:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 48, 735965), True, datetime.datetime(2023, 1, 17, 5, 43, 48, 735965), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:43:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:43:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:39.030814+00:00 (in 50.291920 seconds) repositorygcworker stdout | 2023-01-17 05:43:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:48 UTC)" (scheduled at 2023-01-17 05:43:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:43:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:43:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:43:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:43:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:24.641816+00:00 (in 35.811063 seconds) exportactionlogsworker stdout | 2023-01-17 05:43:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:48 UTC)" (scheduled at 2023-01-17 05:43:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:43:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:43:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 48, 831186), True, datetime.datetime(2023, 1, 17, 5, 43, 48, 831186), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:43:48,839 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:43:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:43:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:48 UTC)" executed successfully builder stdout | 2023-01-17 05:43:52,445 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:52,454 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:52,454 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:52,460 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:52,626 [243] [DEBUG] [app] Starting request: urn:request:2303fd0b-28e4-4d2f-853c-0fb2f136cc9c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:52,626 [239] [DEBUG] [app] Starting request: urn:request:a7cd98c0-38c3-438e-9ae5-7bca383877fb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:43:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:52,631 [232] [DEBUG] [app] Starting request: urn:request:b7c10051-bc59-487b-a5d6-f11f785767d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,631 [242] [DEBUG] [app] Starting request: urn:request:cb71d097-2239-423b-846e-861715f5bfeb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,631 [232] [DEBUG] [app] Ending request: urn:request:b7c10051-bc59-487b-a5d6-f11f785767d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b7c10051-bc59-487b-a5d6-f11f785767d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,631 [242] [DEBUG] [app] Ending request: urn:request:cb71d097-2239-423b-846e-861715f5bfeb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb71d097-2239-423b-846e-861715f5bfeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:43:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:43:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:52,637 [244] [DEBUG] [app] Starting request: urn:request:9119ceea-face-415d-b89e-3a535727793f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:52,637 [244] [DEBUG] [app] Ending request: urn:request:9119ceea-face-415d-b89e-3a535727793f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9119ceea-face-415d-b89e-3a535727793f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:52,637 [241] [DEBUG] [app] Starting request: urn:request:ff9060c9-08e2-454b-988a-f677bb13f6db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:52,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:52,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:52,638 [241] [DEBUG] [app] Ending request: urn:request:ff9060c9-08e2-454b-988a-f677bb13f6db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff9060c9-08e2-454b-988a-f677bb13f6db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:43:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:52,648 [239] [DEBUG] [app] Ending request: urn:request:a7cd98c0-38c3-438e-9ae5-7bca383877fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7cd98c0-38c3-438e-9ae5-7bca383877fb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:52,648 [243] [DEBUG] [app] Ending request: urn:request:2303fd0b-28e4-4d2f-853c-0fb2f136cc9c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2303fd0b-28e4-4d2f-853c-0fb2f136cc9c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:52,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:52,649 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 05:43:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:43:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:43:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:09.230038+00:00 (in 16.392262 seconds) securityworker stdout | 2023-01-17 05:43:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:22 UTC)" (scheduled at 2023-01-17 05:43:52.837365+00:00) securityworker stdout | 2023-01-17 05:43:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:58.735066+00:00 (in 1.799562 seconds) notificationworker stdout | 2023-01-17 05:43:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:48:56 UTC)" (scheduled at 2023-01-17 05:43:56.935055+00:00) notificationworker stdout | 2023-01-17 05:43:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 43, 56, 935872), 'notification/%']) notificationworker stdout | 2023-01-17 05:43:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 43, 56, 935872), True, datetime.datetime(2023, 1, 17, 5, 43, 56, 935872), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:43:56,946 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 43, 56, 935872), True, datetime.datetime(2023, 1, 17, 5, 43, 56, 935872), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 43, 56, 935872), 'notification/%']) notificationworker stdout | 2023-01-17 05:43:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:48:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:43:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:43:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:27.226424+00:00 (in 29.999513 seconds) buildlogsarchiver stdout | 2023-01-17 05:43:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:27 UTC)" (scheduled at 2023-01-17 05:43:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:43:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 43, 57, 227301), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:43:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:43:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:43:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:08.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 05:43:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:08 UTC)" (scheduled at 2023-01-17 05:43:58.735066+00:00) notificationworker stdout | 2023-01-17 05:43:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 58, 735967), True, datetime.datetime(2023, 1, 17, 5, 43, 58, 735967), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:43:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:43:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:15.931994+00:00 (in 16.594152 seconds) namespacegcworker stdout | 2023-01-17 05:43:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:59 UTC)" (scheduled at 2023-01-17 05:43:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:43:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:43:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 59, 338235), True, datetime.datetime(2023, 1, 17, 5, 43, 59, 338235), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:43:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:43:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:43:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:01,113 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:44:01,274 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:44:01,555 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:44:02,349 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:44:02,447 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:44:02,470 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:02,479 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:02,479 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:02,485 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:44:03,682 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:44:04,384 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:44:04,545 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:44:04,846 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:44:05,800 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:44:05,812 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:44:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:44:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:35.848675+00:00 (in 29.999539 seconds) gcworker stdout | 2023-01-17 05:44:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:35 UTC)" (scheduled at 2023-01-17 05:44:05.848675+00:00) gcworker stdout | 2023-01-17 05:44:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724645849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:44:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:44:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:44:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:44:06,114 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:44:06,117 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:44:06,119 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:44:06,706 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:44:07,506 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:44:07,515 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:44:07,626 [244] [DEBUG] [app] Starting request: urn:request:c6511758-cd41-4028-9795-b5306efb43f1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:07,626 [243] [DEBUG] [app] Starting request: urn:request:d23b10f9-bae2-4b4e-b97e-8462d37747c5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:07,630 [242] [DEBUG] [app] Starting request: urn:request:924676f3-b28b-4fad-a60e-9abb658df1ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:07,631 [242] [DEBUG] [app] Ending request: urn:request:924676f3-b28b-4fad-a60e-9abb658df1ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:924676f3-b28b-4fad-a60e-9abb658df1ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:44:07,631 [232] [DEBUG] [app] Starting request: urn:request:9c832f0c-bba7-4fac-9c82-0ef3d633b881 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:07,631 [232] [DEBUG] [app] Ending request: urn:request:9c832f0c-bba7-4fac-9c82-0ef3d633b881 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c832f0c-bba7-4fac-9c82-0ef3d633b881', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:44:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:07,636 [241] [DEBUG] [app] Starting request: urn:request:1992120a-c3b3-4082-920a-e0bbc3a4bfc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:07,636 [241] [DEBUG] [app] Ending request: urn:request:1992120a-c3b3-4082-920a-e0bbc3a4bfc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1992120a-c3b3-4082-920a-e0bbc3a4bfc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:07,636 [239] [DEBUG] [app] Starting request: urn:request:751a6e5a-1795-4535-9d9b-b4eddbb0780e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:07,637 [239] [DEBUG] [app] Ending request: urn:request:751a6e5a-1795-4535-9d9b-b4eddbb0780e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:751a6e5a-1795-4535-9d9b-b4eddbb0780e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:07,647 [244] [DEBUG] [app] Ending request: urn:request:c6511758-cd41-4028-9795-b5306efb43f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c6511758-cd41-4028-9795-b5306efb43f1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:07,647 [243] [DEBUG] [app] Ending request: urn:request:d23b10f9-bae2-4b4e-b97e-8462d37747c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d23b10f9-bae2-4b4e-b97e-8462d37747c5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:44:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-registry stdout | 2023-01-17 05:44:08,498 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:44:08,500 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:44:08,503 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:44:08,506 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:44:08,508 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:44:08,510 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:44:08,513 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:44:08,516 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:44:08,518 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:44:08,652 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:44:08,654 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:44:08,656 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:44:08,659 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:44:08,661 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:44:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:18.735066+00:00 (in 9.999605 seconds) notificationworker stdout | 2023-01-17 05:44:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:18 UTC)" (scheduled at 2023-01-17 05:44:08.735066+00:00) notificationworker stdout | 2023-01-17 05:44:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. storagereplication stdout | 2023-01-17 05:44:08,735 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:44:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 8, 735843), True, datetime.datetime(2023, 1, 17, 5, 44, 8, 735843), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:18 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:22.837365+00:00 (in 13.606851 seconds) securityworker stdout | 2023-01-17 05:44:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:39 UTC)" (scheduled at 2023-01-17 05:44:09.230038+00:00) securityworker stdout | 2023-01-17 05:44:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:44:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:44:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:44:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:44:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:44:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:44:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:09,265 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:44:10,084 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:44:10,133 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:44:10,135 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:44:10,169 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:44:12,495 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:12,504 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:12,504 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:12,510 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:44:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:44:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:47.932490+00:00 (in 32.000043 seconds) namespacegcworker stdout | 2023-01-17 05:44:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:49:15 UTC)" (scheduled at 2023-01-17 05:44:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:44:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 44, 15, 932839), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:44:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 44, 15, 932839), True, datetime.datetime(2023, 1, 17, 5, 44, 15, 932839), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:44:15,943 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 44, 15, 932839), True, datetime.datetime(2023, 1, 17, 5, 44, 15, 932839), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 44, 15, 932839), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:44:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:44:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:49:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:28.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 05:44:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:28 UTC)" (scheduled at 2023-01-17 05:44:18.735066+00:00) notificationworker stdout | 2023-01-17 05:44:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 18, 735904), True, datetime.datetime(2023, 1, 17, 5, 44, 18, 735904), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:44:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:44:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:25.334138+00:00 (in 5.091436 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:44:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:20 UTC)" (scheduled at 2023-01-17 05:44:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:44:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:44:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 20, 243107), True, datetime.datetime(2023, 1, 17, 5, 44, 20, 243107), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:44:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:44:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:44:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:20 UTC)" executed successfully builder stdout | 2023-01-17 05:44:22,511 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:22,519 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:22,519 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:22,525 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:22,625 [244] [DEBUG] [app] Starting request: urn:request:23809356-43d5-40c1-adbe-61f484b5de3b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:22,625 [241] [DEBUG] [app] Starting request: urn:request:a4cbf3ec-7bcc-400e-8a67-3818f29d88a1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:22,630 [242] [DEBUG] [app] Starting request: urn:request:8b6bae51-02ce-49e3-baab-511e1e63ad96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:22,630 [240] [DEBUG] [app] Starting request: urn:request:750c7c09-0c6d-488e-be3c-1c78b8ee1fd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:22,630 [242] [DEBUG] [app] Ending request: urn:request:8b6bae51-02ce-49e3-baab-511e1e63ad96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b6bae51-02ce-49e3-baab-511e1e63ad96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:44:22,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:22,631 [240] [DEBUG] [app] Ending request: urn:request:750c7c09-0c6d-488e-be3c-1c78b8ee1fd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:750c7c09-0c6d-488e-be3c-1c78b8ee1fd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:22,630 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:44:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:22,635 [239] [DEBUG] [app] Starting request: urn:request:39d0ed73-5449-4612-a1c6-2dd4db6a3660 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:22,635 [241] [DEBUG] [app] Starting request: urn:request:1814002d-2392-46bc-b2cf-1a6d54d01c42 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:22,635 [239] [DEBUG] [app] Ending request: urn:request:39d0ed73-5449-4612-a1c6-2dd4db6a3660 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39d0ed73-5449-4612-a1c6-2dd4db6a3660', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:22,635 [241] [DEBUG] [app] Ending request: urn:request:1814002d-2392-46bc-b2cf-1a6d54d01c42 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1814002d-2392-46bc-b2cf-1a6d54d01c42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:22,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:22,635 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:22,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:22,640 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:22,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:22,646 [241] [DEBUG] [app] Ending request: urn:request:a4cbf3ec-7bcc-400e-8a67-3818f29d88a1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a4cbf3ec-7bcc-400e-8a67-3818f29d88a1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:22,646 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:22,646 [244] [DEBUG] [app] Ending request: urn:request:23809356-43d5-40c1-adbe-61f484b5de3b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23809356-43d5-40c1-adbe-61f484b5de3b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:22,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:22,646 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:44:22,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:44:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:39.230038+00:00 (in 16.392227 seconds) securityworker stdout | 2023-01-17 05:44:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:52 UTC)" (scheduled at 2023-01-17 05:44:22.837365+00:00) securityworker stdout | 2023-01-17 05:44:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:22,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:22,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:22,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:44:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:44:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:48.830274+00:00 (in 24.187998 seconds) exportactionlogsworker stdout | 2023-01-17 05:44:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:24 UTC)" (scheduled at 2023-01-17 05:44:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:44:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:44:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:44:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:44:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:20.242198+00:00 (in 54.907564 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:44:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:25 UTC)" (scheduled at 2023-01-17 05:44:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:44:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:44:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:44:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:44:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:57.226424+00:00 (in 29.999511 seconds) buildlogsarchiver stdout | 2023-01-17 05:44:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:57 UTC)" (scheduled at 2023-01-17 05:44:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:44:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 44, 27, 227709), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:44:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:44:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:44:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:38.735066+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:44:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:38 UTC)" (scheduled at 2023-01-17 05:44:28.735066+00:00) notificationworker stdout | 2023-01-17 05:44:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 28, 735948), True, datetime.datetime(2023, 1, 17, 5, 44, 28, 735948), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:31,149 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:44:31,280 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:44:31,590 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:44:32,382 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:44:32,482 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:44:32,534 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:32,543 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:32,543 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:32,549 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:44:33,716 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:44:34,420 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:44:34,566 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:44:34,882 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} globalpromstats stdout | 2023-01-17 05:44:35,833 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:44:35,840 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gcworker stdout | 2023-01-17 05:44:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:44:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:05.848675+00:00 (in 29.999496 seconds) gcworker stdout | 2023-01-17 05:44:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:05 UTC)" (scheduled at 2023-01-17 05:44:35.848675+00:00) gcworker stdout | 2023-01-17 05:44:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724675849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:44:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:44:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:44:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:44:36,121 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:44:36,123 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:44:36,126 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:44:36,741 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:44:37,532 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:44:37,550 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:44:37,625 [239] [DEBUG] [app] Starting request: urn:request:acf836d9-9545-4cc2-807d-4bf00561d8d1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:37,626 [241] [DEBUG] [app] Starting request: urn:request:401790cd-fd6f-47f7-a6dd-aec95846e439 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:37,630 [242] [DEBUG] [app] Starting request: urn:request:30e481f7-055c-4b02-af99-948b4a045988 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:37,631 [242] [DEBUG] [app] Ending request: urn:request:30e481f7-055c-4b02-af99-948b4a045988 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30e481f7-055c-4b02-af99-948b4a045988', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:44:37,631 [236] [DEBUG] [app] Starting request: urn:request:688952e8-8616-4967-9f2c-899c4e36d2c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:44:37,631 [236] [DEBUG] [app] Ending request: urn:request:688952e8-8616-4967-9f2c-899c4e36d2c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:688952e8-8616-4967-9f2c-899c4e36d2c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:44:37,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:37,636 [244] [DEBUG] [app] Starting request: urn:request:0a05aa2e-5261-4e42-86d4-8e4b95d8d2d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:37,636 [244] [DEBUG] [app] Ending request: urn:request:0a05aa2e-5261-4e42-86d4-8e4b95d8d2d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a05aa2e-5261-4e42-86d4-8e4b95d8d2d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:37,636 [239] [DEBUG] [app] Starting request: urn:request:5009b844-7ef1-42e7-b3c3-c8a564af7fa3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:37,637 [239] [DEBUG] [app] Ending request: urn:request:5009b844-7ef1-42e7-b3c3-c8a564af7fa3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5009b844-7ef1-42e7-b3c3-c8a564af7fa3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:37,647 [241] [DEBUG] [app] Ending request: urn:request:401790cd-fd6f-47f7-a6dd-aec95846e439 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:401790cd-fd6f-47f7-a6dd-aec95846e439', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:37,648 [239] [DEBUG] [app] Ending request: urn:request:acf836d9-9545-4cc2-807d-4bf00561d8d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:acf836d9-9545-4cc2-807d-4bf00561d8d1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:44:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-registry stdout | 2023-01-17 05:44:38,506 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:44:38,508 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:44:38,514 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:44:38,516 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:44:38,519 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:44:38,521 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:44:38,523 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:44:38,526 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:44:38,528 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:44:38,657 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:44:38,660 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:44:38,663 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:44:38,666 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:44:38,668 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:44:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:43.935629+00:00 (in 5.200113 seconds) notificationworker stdout | 2023-01-17 05:44:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:48 UTC)" (scheduled at 2023-01-17 05:44:38.735066+00:00) notificationworker stdout | 2023-01-17 05:44:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 38, 735870), True, datetime.datetime(2023, 1, 17, 5, 44, 38, 735870), 0, 'notification/%', 50, 1, 0]) storagereplication stdout | 2023-01-17 05:44:38,741 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:44:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:44:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:44:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:48.738458+00:00 (in 9.707191 seconds) repositorygcworker stdout | 2023-01-17 05:44:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:39 UTC)" (scheduled at 2023-01-17 05:44:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:44:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:44:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 39, 31703), True, datetime.datetime(2023, 1, 17, 5, 44, 39, 31703), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:44:39,045 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:44:39,045 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:44:39,045 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:52.837365+00:00 (in 13.606890 seconds) securityworker stdout | 2023-01-17 05:44:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:09 UTC)" (scheduled at 2023-01-17 05:44:39.230038+00:00) securityworker stdout | 2023-01-17 05:44:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:44:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:44:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:44:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:39,271 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:44:40,109 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:44:40,144 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:44:40,147 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:44:40,204 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:44:42,559 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:42,568 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:42,568 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:42,574 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:48.735066+00:00 (in 4.798991 seconds) notificationworker stdout | 2023-01-17 05:44:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:43 UTC)" (scheduled at 2023-01-17 05:44:43.935629+00:00) notificationworker stdout | 2023-01-17 05:44:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:44:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:44:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:44:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:59.337413+00:00 (in 11.404425 seconds) namespacegcworker stdout | 2023-01-17 05:44:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:47 UTC)" (scheduled at 2023-01-17 05:44:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:44:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:44:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:58.735066+00:00 (in 9.999491 seconds) notificationworker stdout | 2023-01-17 05:44:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:58 UTC)" (scheduled at 2023-01-17 05:44:48.735066+00:00) notificationworker stdout | 2023-01-17 05:44:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 48, 735988), True, datetime.datetime(2023, 1, 17, 5, 44, 48, 735988), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:44:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:44:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:39.030814+00:00 (in 50.291902 seconds) repositorygcworker stdout | 2023-01-17 05:44:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:48 UTC)" (scheduled at 2023-01-17 05:44:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:44:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:44:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:44:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:44:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:24.641816+00:00 (in 35.811079 seconds) exportactionlogsworker stdout | 2023-01-17 05:44:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:48 UTC)" (scheduled at 2023-01-17 05:44:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:44:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:44:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 48, 831127), True, datetime.datetime(2023, 1, 17, 5, 44, 48, 831127), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:44:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:44:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:44:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:48 UTC)" executed successfully builder stdout | 2023-01-17 05:44:52,583 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:52,592 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:52,592 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:52,598 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:52,625 [243] [DEBUG] [app] Starting request: urn:request:9fa91589-56ef-4a54-8c72-f9bae4f1d7d8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:52,625 [241] [DEBUG] [app] Starting request: urn:request:26555e5d-d7e0-4e69-ba3b-a7c204176976 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:44:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:52,629 [232] [DEBUG] [app] Starting request: urn:request:bec979f6-2824-45cb-824d-f918508140c4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:52,630 [242] [DEBUG] [app] Starting request: urn:request:c2fa9ac8-0d75-400f-894a-7d0465911698 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:52,630 [232] [DEBUG] [app] Ending request: urn:request:bec979f6-2824-45cb-824d-f918508140c4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bec979f6-2824-45cb-824d-f918508140c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:52,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:44:52,630 [242] [DEBUG] [app] Ending request: urn:request:c2fa9ac8-0d75-400f-894a-7d0465911698 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2fa9ac8-0d75-400f-894a-7d0465911698', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:52,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:44:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:52,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:52,634 [244] [DEBUG] [app] Starting request: urn:request:00990b97-f9ff-4755-8eae-30ddbf2c00a0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:52,635 [244] [DEBUG] [app] Ending request: urn:request:00990b97-f9ff-4755-8eae-30ddbf2c00a0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00990b97-f9ff-4755-8eae-30ddbf2c00a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:52,635 [239] [DEBUG] [app] Starting request: urn:request:36576a71-2fef-4d32-8886-c661e55b51e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:52,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:52,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:52,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:52,636 [239] [DEBUG] [app] Ending request: urn:request:36576a71-2fef-4d32-8886-c661e55b51e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36576a71-2fef-4d32-8886-c661e55b51e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:52,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:52,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:52,645 [243] [DEBUG] [app] Ending request: urn:request:9fa91589-56ef-4a54-8c72-f9bae4f1d7d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9fa91589-56ef-4a54-8c72-f9bae4f1d7d8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:52,646 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:44:52,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:44:52,646 [241] [DEBUG] [app] Ending request: urn:request:26555e5d-d7e0-4e69-ba3b-a7c204176976 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26555e5d-d7e0-4e69-ba3b-a7c204176976', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:52,646 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:52,646 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) securityworker stdout | 2023-01-17 05:44:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:09.230038+00:00 (in 16.392267 seconds) securityworker stdout | 2023-01-17 05:44:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:22 UTC)" (scheduled at 2023-01-17 05:44:52.837365+00:00) securityworker stdout | 2023-01-17 05:44:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:44:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:44:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:27.226424+00:00 (in 29.999575 seconds) buildlogsarchiver stdout | 2023-01-17 05:44:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:27 UTC)" (scheduled at 2023-01-17 05:44:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:44:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 44, 57, 227209), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:44:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:44:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:44:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:08.735066+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:44:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:08 UTC)" (scheduled at 2023-01-17 05:44:58.735066+00:00) notificationworker stdout | 2023-01-17 05:44:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 58, 735975), True, datetime.datetime(2023, 1, 17, 5, 44, 58, 735975), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:58,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:44:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:44:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:47.932490+00:00 (in 48.594603 seconds) namespacegcworker stdout | 2023-01-17 05:44:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:59 UTC)" (scheduled at 2023-01-17 05:44:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:44:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:44:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 59, 338284), True, datetime.datetime(2023, 1, 17, 5, 44, 59, 338284), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:44:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:44:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:44:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:01,184 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:45:01,311 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:45:01,625 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:45:02,417 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:45:02,487 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:45:02,608 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:02,617 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:02,617 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:02,623 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:45:03,736 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:45:04,455 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:45:04,600 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:45:04,891 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:45:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:45:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:35.848675+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 05:45:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:35 UTC)" (scheduled at 2023-01-17 05:45:05.848675+00:00) gcworker stdout | 2023-01-17 05:45:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724705849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:45:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:45:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:45:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:45:05,868 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:45:05,874 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:45:06,128 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:45:06,131 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:45:06,133 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:45:06,776 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:45:07,568 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:45:07,585 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:45:07,625 [244] [DEBUG] [app] Starting request: urn:request:3d3c71cf-93e8-47a5-a706-45abb8ba8bca (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:07,626 [239] [DEBUG] [app] Starting request: urn:request:b10f6eea-0374-4bc6-b66f-50286abf849a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:07,630 [232] [DEBUG] [app] Starting request: urn:request:feb12572-58ee-4e68-bc4f-f0884df251ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:07,631 [240] [DEBUG] [app] Starting request: urn:request:aac1c934-4abc-426c-b742-becc3c82caaf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:07,631 [232] [DEBUG] [app] Ending request: urn:request:feb12572-58ee-4e68-bc4f-f0884df251ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:feb12572-58ee-4e68-bc4f-f0884df251ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:45:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:45:07,631 [240] [DEBUG] [app] Ending request: urn:request:aac1c934-4abc-426c-b742-becc3c82caaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aac1c934-4abc-426c-b742-becc3c82caaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:45:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:45:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:07,635 [243] [DEBUG] [app] Starting request: urn:request:227b95b1-d41d-46aa-a002-0452a2749660 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:07,636 [244] [DEBUG] [app] Starting request: urn:request:9c454a2b-59be-4046-ad0c-46ae7646ef82 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:07,636 [243] [DEBUG] [app] Ending request: urn:request:227b95b1-d41d-46aa-a002-0452a2749660 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:227b95b1-d41d-46aa-a002-0452a2749660', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:07,636 [244] [DEBUG] [app] Ending request: urn:request:9c454a2b-59be-4046-ad0c-46ae7646ef82 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9c454a2b-59be-4046-ad0c-46ae7646ef82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:45:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:07,647 [239] [DEBUG] [app] Ending request: urn:request:b10f6eea-0374-4bc6-b66f-50286abf849a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b10f6eea-0374-4bc6-b66f-50286abf849a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:07,647 [244] [DEBUG] [app] Ending request: urn:request:3d3c71cf-93e8-47a5-a706-45abb8ba8bca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d3c71cf-93e8-47a5-a706-45abb8ba8bca', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:45:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:45:08,513 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:45:08,518 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:45:08,524 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:45:08,527 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:45:08,529 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:45:08,531 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:45:08,534 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:45:08,536 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:45:08,538 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:45:08,664 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:45:08,667 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:45:08,669 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:45:08,672 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:45:08,674 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:45:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:18.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 05:45:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:18 UTC)" (scheduled at 2023-01-17 05:45:08.735066+00:00) notificationworker stdout | 2023-01-17 05:45:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 8, 735940), True, datetime.datetime(2023, 1, 17, 5, 45, 8, 735940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:18 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:45:08,776 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:45:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:22.837365+00:00 (in 13.606850 seconds) securityworker stdout | 2023-01-17 05:45:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:39 UTC)" (scheduled at 2023-01-17 05:45:09.230038+00:00) securityworker stdout | 2023-01-17 05:45:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:45:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:45:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:45:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:45:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:45:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:45:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:09,308 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:45:10,140 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:45:10,164 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:45:10,178 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:45:10,239 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:45:12,633 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:12,642 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:12,642 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:12,648 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:28.735066+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:45:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:28 UTC)" (scheduled at 2023-01-17 05:45:18.735066+00:00) notificationworker stdout | 2023-01-17 05:45:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 18, 735910), True, datetime.datetime(2023, 1, 17, 5, 45, 18, 735910), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:45:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:45:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:25.334138+00:00 (in 5.091485 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:45:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:20 UTC)" (scheduled at 2023-01-17 05:45:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:45:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:45:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 20, 243071), True, datetime.datetime(2023, 1, 17, 5, 45, 20, 243071), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:45:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:45:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:45:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:45:22,626 [239] [DEBUG] [app] Starting request: urn:request:a8939e93-90b4-402c-8d25-7bb314e522dc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:22,626 [243] [DEBUG] [app] Starting request: urn:request:43a75ef3-2a88-452d-9a07-c599104c54dc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:22,630 [242] [DEBUG] [app] Starting request: urn:request:4b7c5425-65f0-4c2c-91bf-fa526caddc95 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,631 [237] [DEBUG] [app] Starting request: urn:request:cdb85f18-f0d0-41fd-a480-8ad3996cddb1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,631 [242] [DEBUG] [app] Ending request: urn:request:4b7c5425-65f0-4c2c-91bf-fa526caddc95 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b7c5425-65f0-4c2c-91bf-fa526caddc95', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:45:22,631 [237] [DEBUG] [app] Ending request: urn:request:cdb85f18-f0d0-41fd-a480-8ad3996cddb1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdb85f18-f0d0-41fd-a480-8ad3996cddb1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:45:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:45:22,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:22,636 [241] [DEBUG] [app] Starting request: urn:request:f1caed31-b817-4b10-aedd-b54782b4919b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:22,636 [241] [DEBUG] [app] Ending request: urn:request:f1caed31-b817-4b10-aedd-b54782b4919b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f1caed31-b817-4b10-aedd-b54782b4919b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:22,636 [244] [DEBUG] [app] Starting request: urn:request:6fc8d0d3-15d4-4787-aee7-4ffb7bac98c6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:22,636 [244] [DEBUG] [app] Ending request: urn:request:6fc8d0d3-15d4-4787-aee7-4ffb7bac98c6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6fc8d0d3-15d4-4787-aee7-4ffb7bac98c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:22,647 [239] [DEBUG] [app] Ending request: urn:request:a8939e93-90b4-402c-8d25-7bb314e522dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8939e93-90b4-402c-8d25-7bb314e522dc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:22,647 [243] [DEBUG] [app] Ending request: urn:request:43a75ef3-2a88-452d-9a07-c599104c54dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43a75ef3-2a88-452d-9a07-c599104c54dc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:45:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) builder stdout | 2023-01-17 05:45:22,651 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:22,658 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:22,658 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:22,664 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:39.230038+00:00 (in 16.392191 seconds) securityworker stdout | 2023-01-17 05:45:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:52 UTC)" (scheduled at 2023-01-17 05:45:22.837365+00:00) securityworker stdout | 2023-01-17 05:45:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:45:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:45:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:48.830274+00:00 (in 24.187996 seconds) exportactionlogsworker stdout | 2023-01-17 05:45:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:24 UTC)" (scheduled at 2023-01-17 05:45:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:45:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:45:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:45:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:45:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:20.242198+00:00 (in 54.907558 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:45:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:25 UTC)" (scheduled at 2023-01-17 05:45:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:45:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:45:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:45:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:45:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:57.226424+00:00 (in 29.999537 seconds) buildlogsarchiver stdout | 2023-01-17 05:45:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:57 UTC)" (scheduled at 2023-01-17 05:45:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:45:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 45, 27, 227261), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:45:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:45:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:45:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:38.735066+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:45:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:38 UTC)" (scheduled at 2023-01-17 05:45:28.735066+00:00) notificationworker stdout | 2023-01-17 05:45:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 28, 736013), True, datetime.datetime(2023, 1, 17, 5, 45, 28, 736013), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:31,210 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:45:31,346 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:45:31,652 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:45:32,451 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:45:32,513 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:45:32,674 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:32,683 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:32,683 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:32,689 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:45:33,745 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:45:34,486 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:45:34,635 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:45:34,899 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:45:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:45:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:05.848675+00:00 (in 29.999532 seconds) gcworker stdout | 2023-01-17 05:45:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:05 UTC)" (scheduled at 2023-01-17 05:45:35.848675+00:00) gcworker stdout | 2023-01-17 05:45:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724735849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:45:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:45:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:45:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:05 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:45:35,904 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:45:35,908 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:45:36,135 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:45:36,137 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:45:36,139 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:45:36,812 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:45:37,584 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:45:37,614 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:45:37,625 [243] [DEBUG] [app] Starting request: urn:request:9b47ea4a-1fb1-4eea-8625-86a3700bb468 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:37,625 [239] [DEBUG] [app] Starting request: urn:request:c453cf56-3229-4cf4-a3df-c3e1582e9b6d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:37,630 [232] [DEBUG] [app] Starting request: urn:request:77008e95-2c78-4e98-a554-696660cee1b5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:37,631 [242] [DEBUG] [app] Starting request: urn:request:f58ccb79-2625-4957-80db-3cc69ee38703 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:37,631 [232] [DEBUG] [app] Ending request: urn:request:77008e95-2c78-4e98-a554-696660cee1b5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77008e95-2c78-4e98-a554-696660cee1b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:45:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:45:37,631 [242] [DEBUG] [app] Ending request: urn:request:f58ccb79-2625-4957-80db-3cc69ee38703 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f58ccb79-2625-4957-80db-3cc69ee38703', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:45:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:45:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:37,636 [241] [DEBUG] [app] Starting request: urn:request:e6106fba-ea36-4306-9b72-61e06402b0c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:37,636 [241] [DEBUG] [app] Ending request: urn:request:e6106fba-ea36-4306-9b72-61e06402b0c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6106fba-ea36-4306-9b72-61e06402b0c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:37,636 [244] [DEBUG] [app] Starting request: urn:request:833d00bd-a800-4f4b-b08f-9785df7e2032 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:37,637 [244] [DEBUG] [app] Ending request: urn:request:833d00bd-a800-4f4b-b08f-9785df7e2032 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:833d00bd-a800-4f4b-b08f-9785df7e2032', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:45:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:37,647 [239] [DEBUG] [app] Ending request: urn:request:c453cf56-3229-4cf4-a3df-c3e1582e9b6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c453cf56-3229-4cf4-a3df-c3e1582e9b6d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:37,647 [243] [DEBUG] [app] Ending request: urn:request:9b47ea4a-1fb1-4eea-8625-86a3700bb468 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b47ea4a-1fb1-4eea-8625-86a3700bb468', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:45:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-registry stdout | 2023-01-17 05:45:38,521 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:45:38,528 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:45:38,533 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:45:38,535 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:45:38,537 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:45:38,540 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:45:38,542 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:45:38,545 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:45:38,548 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:45:38,670 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:45:38,673 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:45:38,676 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:45:38,678 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:45:38,680 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:45:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:43.935629+00:00 (in 5.200091 seconds) notificationworker stdout | 2023-01-17 05:45:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:48 UTC)" (scheduled at 2023-01-17 05:45:38.735066+00:00) notificationworker stdout | 2023-01-17 05:45:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 38, 735901), True, datetime.datetime(2023, 1, 17, 5, 45, 38, 735901), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:48 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:45:38,786 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:45:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:45:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:48.738458+00:00 (in 9.707243 seconds) repositorygcworker stdout | 2023-01-17 05:45:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:39 UTC)" (scheduled at 2023-01-17 05:45:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:45:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:45:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 39, 31615), True, datetime.datetime(2023, 1, 17, 5, 45, 39, 31615), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:45:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:45:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:45:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:52.837365+00:00 (in 13.606829 seconds) securityworker stdout | 2023-01-17 05:45:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:09 UTC)" (scheduled at 2023-01-17 05:45:39.230038+00:00) securityworker stdout | 2023-01-17 05:45:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:45:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:45:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:39,235 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:39,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:45:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:39,338 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:45:40,167 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:45:40,182 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:45:40,213 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:45:40,272 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:45:42,692 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:42,701 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:42,701 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:42,707 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:48.735066+00:00 (in 4.799002 seconds) notificationworker stdout | 2023-01-17 05:45:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:43 UTC)" (scheduled at 2023-01-17 05:45:43.935629+00:00) notificationworker stdout | 2023-01-17 05:45:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:45:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:45:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:45:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:59.337413+00:00 (in 11.404432 seconds) namespacegcworker stdout | 2023-01-17 05:45:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:47 UTC)" (scheduled at 2023-01-17 05:45:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:45:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:45:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:58.735066+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 05:45:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:58 UTC)" (scheduled at 2023-01-17 05:45:48.735066+00:00) notificationworker stdout | 2023-01-17 05:45:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 48, 735864), True, datetime.datetime(2023, 1, 17, 5, 45, 48, 735864), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:45:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:45:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:39.030814+00:00 (in 50.291917 seconds) repositorygcworker stdout | 2023-01-17 05:45:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:48 UTC)" (scheduled at 2023-01-17 05:45:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:45:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:45:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:45:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:45:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:24.641816+00:00 (in 35.811058 seconds) exportactionlogsworker stdout | 2023-01-17 05:45:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:48 UTC)" (scheduled at 2023-01-17 05:45:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:45:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:45:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 48, 831144), True, datetime.datetime(2023, 1, 17, 5, 45, 48, 831144), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:45:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:45:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:45:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:45:52,626 [243] [DEBUG] [app] Starting request: urn:request:efe022c1-a084-48f5-993f-4995e469d788 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:52,626 [241] [DEBUG] [app] Starting request: urn:request:e89a1963-3dc2-45b6-bb76-b6c8ddcc55d5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:45:52,628 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:52,631 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:52,632 [240] [DEBUG] [app] Starting request: urn:request:4f3213f3-23c0-4ba2-a6fe-199ae08a649b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:52,632 [242] [DEBUG] [app] Starting request: urn:request:4d04e68b-745e-443e-bb21-691e0e7f6693 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:52,632 [240] [DEBUG] [app] Ending request: urn:request:4f3213f3-23c0-4ba2-a6fe-199ae08a649b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f3213f3-23c0-4ba2-a6fe-199ae08a649b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:45:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:45:52,632 [242] [DEBUG] [app] Ending request: urn:request:4d04e68b-745e-443e-bb21-691e0e7f6693 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d04e68b-745e-443e-bb21-691e0e7f6693', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:52,633 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:45:52,633 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:45:52,633 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:52,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:52,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:52,637 [239] [DEBUG] [app] Starting request: urn:request:d5752b02-821d-4c72-9f3f-802c01deea6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:52,637 [244] [DEBUG] [app] Starting request: urn:request:10283865-f06b-4cce-aa09-528d1623ef98 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:52,637 [239] [DEBUG] [app] Ending request: urn:request:d5752b02-821d-4c72-9f3f-802c01deea6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5752b02-821d-4c72-9f3f-802c01deea6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:45:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:52,637 [244] [DEBUG] [app] Ending request: urn:request:10283865-f06b-4cce-aa09-528d1623ef98 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10283865-f06b-4cce-aa09-528d1623ef98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:52,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:52,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:52,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:52,649 [241] [DEBUG] [app] Ending request: urn:request:e89a1963-3dc2-45b6-bb76-b6c8ddcc55d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e89a1963-3dc2-45b6-bb76-b6c8ddcc55d5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:52,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:52,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:45:52,653 [243] [DEBUG] [app] Ending request: urn:request:efe022c1-a084-48f5-993f-4995e469d788 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:efe022c1-a084-48f5-993f-4995e469d788', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:52,653 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:45:52,653 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:45:52,718 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:52,726 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:52,726 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:52,732 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:09.230038+00:00 (in 16.392143 seconds) securityworker stdout | 2023-01-17 05:45:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:22 UTC)" (scheduled at 2023-01-17 05:45:52.837365+00:00) securityworker stdout | 2023-01-17 05:45:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:45:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:45:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:27.226424+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 05:45:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:27 UTC)" (scheduled at 2023-01-17 05:45:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:45:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 45, 57, 227234), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:45:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:45:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:45:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:08.735066+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 05:45:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:08 UTC)" (scheduled at 2023-01-17 05:45:58.735066+00:00) notificationworker stdout | 2023-01-17 05:45:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 58, 735939), True, datetime.datetime(2023, 1, 17, 5, 45, 58, 735939), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:58,746 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:58,746 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:58,746 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:45:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:45:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:47.932490+00:00 (in 48.594628 seconds) namespacegcworker stdout | 2023-01-17 05:45:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:59 UTC)" (scheduled at 2023-01-17 05:45:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:45:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:45:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 59, 338234), True, datetime.datetime(2023, 1, 17, 5, 45, 59, 338234), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:45:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:45:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:45:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:01,220 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:46:01,365 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:46:01,687 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:46:02,469 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:46:02,526 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:46:02,742 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:02,751 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:02,751 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:02,757 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:46:03,781 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:46:04,522 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:46:04,642 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:46:04,908 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:46:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:46:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:35.848675+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 05:46:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:35 UTC)" (scheduled at 2023-01-17 05:46:05.848675+00:00) gcworker stdout | 2023-01-17 05:46:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724765849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:46:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:46:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:46:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:46:05,934 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:46:05,943 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:46:06,141 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:46:06,144 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:46:06,146 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:46:06,839 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 05:46:07,593 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:46:07,625 [239] [DEBUG] [app] Starting request: urn:request:905e2170-e60b-4dcd-bf80-609084ab4bf4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:07,625 [244] [DEBUG] [app] Starting request: urn:request:d91387bc-36e1-44e3-888f-3036fb4eb4a7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:07,630 [242] [DEBUG] [app] Starting request: urn:request:dc927137-136d-4dfd-827e-fc8992e66faf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:07,631 [240] [DEBUG] [app] Starting request: urn:request:7656b5aa-1129-4645-90e8-1e84944059c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:07,631 [242] [DEBUG] [app] Ending request: urn:request:dc927137-136d-4dfd-827e-fc8992e66faf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc927137-136d-4dfd-827e-fc8992e66faf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:46:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:46:07,631 [240] [DEBUG] [app] Ending request: urn:request:7656b5aa-1129-4645-90e8-1e84944059c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7656b5aa-1129-4645-90e8-1e84944059c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:46:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:46:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:07,635 [243] [DEBUG] [app] Starting request: urn:request:15a52b33-cf13-41cf-9599-82d8509560d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:07,636 [244] [DEBUG] [app] Starting request: urn:request:4181f276-6512-476d-933c-05c26b1ed1c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:07,636 [243] [DEBUG] [app] Ending request: urn:request:15a52b33-cf13-41cf-9599-82d8509560d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15a52b33-cf13-41cf-9599-82d8509560d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:07,636 [244] [DEBUG] [app] Ending request: urn:request:4181f276-6512-476d-933c-05c26b1ed1c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4181f276-6512-476d-933c-05c26b1ed1c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:07,647 [244] [DEBUG] [app] Ending request: urn:request:d91387bc-36e1-44e3-888f-3036fb4eb4a7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d91387bc-36e1-44e3-888f-3036fb4eb4a7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:07,647 [239] [DEBUG] [app] Ending request: urn:request:905e2170-e60b-4dcd-bf80-609084ab4bf4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:905e2170-e60b-4dcd-bf80-609084ab4bf4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:46:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) queuecleanupworker stdout | 2023-01-17 05:46:07,649 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:46:08,532 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:46:08,538 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:46:08,540 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:46:08,543 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:46:08,545 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:46:08,549 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:46:08,551 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:46:08,554 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:46:08,556 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:46:08,677 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:46:08,679 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:46:08,683 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:46:08,686 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:46:08,688 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:46:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:18.735066+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:46:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:18 UTC)" (scheduled at 2023-01-17 05:46:08.735066+00:00) notificationworker stdout | 2023-01-17 05:46:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 8, 735905), True, datetime.datetime(2023, 1, 17, 5, 46, 8, 735905), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:18 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:46:08,821 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:46:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:22.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 05:46:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:39 UTC)" (scheduled at 2023-01-17 05:46:09.230038+00:00) securityworker stdout | 2023-01-17 05:46:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:46:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:46:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:46:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:09,349 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:46:10,185 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:46:10,216 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:46:10,219 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:46:10,308 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:46:12,767 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:12,776 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:12,776 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:12,782 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:28.735066+00:00 (in 9.999570 seconds) notificationworker stdout | 2023-01-17 05:46:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:28 UTC)" (scheduled at 2023-01-17 05:46:18.735066+00:00) notificationworker stdout | 2023-01-17 05:46:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 18, 735884), True, datetime.datetime(2023, 1, 17, 5, 46, 18, 735884), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:46:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:46:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:25.334138+00:00 (in 5.091476 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:46:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:20 UTC)" (scheduled at 2023-01-17 05:46:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:46:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:46:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 20, 243086), True, datetime.datetime(2023, 1, 17, 5, 46, 20, 243086), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:46:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:46:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:46:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:46:22,625 [239] [DEBUG] [app] Starting request: urn:request:2452ea05-c6e0-4dd0-9ec1-270a2047d71c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:22,625 [244] [DEBUG] [app] Starting request: urn:request:28d14dfc-6c12-4ad4-bf8e-22d49be79861 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:22,630 [242] [DEBUG] [app] Starting request: urn:request:954442a3-349d-405c-a49c-5ada96e261f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:22,631 [242] [DEBUG] [app] Ending request: urn:request:954442a3-349d-405c-a49c-5ada96e261f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:954442a3-349d-405c-a49c-5ada96e261f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:22,631 [237] [DEBUG] [app] Starting request: urn:request:3fa9182d-4ebd-4f32-9c3a-04db9916e08a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:46:22,631 [237] [DEBUG] [app] Ending request: urn:request:3fa9182d-4ebd-4f32-9c3a-04db9916e08a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3fa9182d-4ebd-4f32-9c3a-04db9916e08a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:46:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:46:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:22,636 [244] [DEBUG] [app] Starting request: urn:request:b9a4f95f-052f-438c-a450-452c668db5c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:22,636 [244] [DEBUG] [app] Ending request: urn:request:b9a4f95f-052f-438c-a450-452c668db5c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9a4f95f-052f-438c-a450-452c668db5c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:22,636 [243] [DEBUG] [app] Starting request: urn:request:6acdf5a9-dab6-4fbc-812e-997137634bde (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:22,637 [243] [DEBUG] [app] Ending request: urn:request:6acdf5a9-dab6-4fbc-812e-997137634bde (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6acdf5a9-dab6-4fbc-812e-997137634bde', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:46:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:22,647 [244] [DEBUG] [app] Ending request: urn:request:28d14dfc-6c12-4ad4-bf8e-22d49be79861 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28d14dfc-6c12-4ad4-bf8e-22d49be79861', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:46:22,648 [239] [DEBUG] [app] Ending request: urn:request:2452ea05-c6e0-4dd0-9ec1-270a2047d71c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2452ea05-c6e0-4dd0-9ec1-270a2047d71c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) builder stdout | 2023-01-17 05:46:22,782 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:22,790 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:22,790 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:22,796 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:39.230038+00:00 (in 16.392218 seconds) securityworker stdout | 2023-01-17 05:46:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:52 UTC)" (scheduled at 2023-01-17 05:46:22.837365+00:00) securityworker stdout | 2023-01-17 05:46:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:46:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:46:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:44.641497+00:00 (in 19.999252 seconds) exportactionlogsworker stdout | 2023-01-17 05:46:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:24 UTC)" (scheduled at 2023-01-17 05:46:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:46:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:46:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:46:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:46:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:39.333860+00:00 (in 13.999212 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:46:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:25 UTC)" (scheduled at 2023-01-17 05:46:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:46:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:46:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:46:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:46:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:57.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 05:46:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:57 UTC)" (scheduled at 2023-01-17 05:46:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:46:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 46, 27, 227233), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:46:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:46:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:46:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:38.735066+00:00 (in 9.999572 seconds) notificationworker stdout | 2023-01-17 05:46:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:38 UTC)" (scheduled at 2023-01-17 05:46:28.735066+00:00) notificationworker stdout | 2023-01-17 05:46:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 28, 735867), True, datetime.datetime(2023, 1, 17, 5, 46, 28, 735867), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:31,255 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:46:31,385 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:46:31,717 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:46:32,505 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:46:32,561 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:46:32,806 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:32,815 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:32,815 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:32,821 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:46:33,817 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:46:34,555 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:46:34,674 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:46:34,914 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:46:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:46:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:05.848675+00:00 (in 29.999551 seconds) gcworker stdout | 2023-01-17 05:46:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:05 UTC)" (scheduled at 2023-01-17 05:46:35.848675+00:00) gcworker stdout | 2023-01-17 05:46:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724795849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:46:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:46:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:46:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:05 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:46:35,970 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:46:35,978 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:46:36,149 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:46:36,152 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:46:36,154 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:46:36,875 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:46:37,625 [244] [DEBUG] [app] Starting request: urn:request:8d5c8563-d470-4dc6-9276-ceb57a3cadfb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:37,625 [241] [DEBUG] [app] Starting request: urn:request:25f81396-ad74-464f-ab7a-adabc7b78afc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( logrotateworker stdout | 2023-01-17 05:46:37,629 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:46:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:37,630 [232] [DEBUG] [app] Starting request: urn:request:9da2998f-3cac-42b2-bc99-a800a4a6e0a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:37,630 [240] [DEBUG] [app] Starting request: urn:request:cf14d0bc-98b6-4096-a5ca-d75f064a2ba3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:37,630 [232] [DEBUG] [app] Ending request: urn:request:9da2998f-3cac-42b2-bc99-a800a4a6e0a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9da2998f-3cac-42b2-bc99-a800a4a6e0a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:37,631 [240] [DEBUG] [app] Ending request: urn:request:cf14d0bc-98b6-4096-a5ca-d75f064a2ba3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf14d0bc-98b6-4096-a5ca-d75f064a2ba3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:46:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:46:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:46:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:37,635 [244] [DEBUG] [app] Starting request: urn:request:9d34f4c8-8201-4333-89a6-d58802066086 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:37,636 [244] [DEBUG] [app] Ending request: urn:request:9d34f4c8-8201-4333-89a6-d58802066086 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d34f4c8-8201-4333-89a6-d58802066086', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:37,636 [243] [DEBUG] [app] Starting request: urn:request:0a5db082-b206-43ef-a26d-678f47c1557d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:37,636 [243] [DEBUG] [app] Ending request: urn:request:0a5db082-b206-43ef-a26d-678f47c1557d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a5db082-b206-43ef-a26d-678f47c1557d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:37,647 [244] [DEBUG] [app] Ending request: urn:request:8d5c8563-d470-4dc6-9276-ceb57a3cadfb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d5c8563-d470-4dc6-9276-ceb57a3cadfb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:37,647 [241] [DEBUG] [app] Ending request: urn:request:25f81396-ad74-464f-ab7a-adabc7b78afc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:25f81396-ad74-464f-ab7a-adabc7b78afc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:46:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) queuecleanupworker stdout | 2023-01-17 05:46:37,684 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:46:38,543 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:46:38,546 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:46:38,549 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:46:38,551 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:46:38,555 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:46:38,558 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:46:38,561 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:46:38,563 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:46:38,566 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:46:38,684 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:46:38,686 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:46:38,690 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:46:38,694 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:46:38,696 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:46:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:43.935629+00:00 (in 5.200149 seconds) notificationworker stdout | 2023-01-17 05:46:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:48 UTC)" (scheduled at 2023-01-17 05:46:38.735066+00:00) notificationworker stdout | 2023-01-17 05:46:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 38, 735838), True, datetime.datetime(2023, 1, 17, 5, 46, 38, 735838), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:48 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:46:38,843 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:46:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:46:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:48.738458+00:00 (in 9.707200 seconds) repositorygcworker stdout | 2023-01-17 05:46:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:39 UTC)" (scheduled at 2023-01-17 05:46:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:46:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:46:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 39, 31694), True, datetime.datetime(2023, 1, 17, 5, 46, 39, 31694), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:46:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:46:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:46:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:52.837365+00:00 (in 13.606811 seconds) securityworker stdout | 2023-01-17 05:46:39,238 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:09 UTC)" (scheduled at 2023-01-17 05:46:39.230038+00:00) securityworker stdout | 2023-01-17 05:46:39,238 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:39,238 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:46:39,240 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:39,240 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:46:39,240 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:39,241 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:39,241 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:39,244 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:39,246 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:39,271 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:39,271 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:46:39,271 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:46:39,271 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:46:39,272 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:39,272 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:39,272 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:39,272 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:46:39,272 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:46:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:46:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:20.242198+00:00 (in 40.907869 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:46:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:39 UTC)" (scheduled at 2023-01-17 05:46:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:46:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 46, 39, 334721), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:46:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 39, 334721), True, datetime.datetime(2023, 1, 17, 5, 46, 39, 334721), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:46:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 39, 334721), True, datetime.datetime(2023, 1, 17, 5, 46, 39, 334721), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 46, 39, 334721), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:46:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:46:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:39,385 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:46:40,221 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:46:40,223 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:46:40,242 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:46:40,343 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:46:42,831 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:42,840 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:42,840 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:42,846 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:48.735066+00:00 (in 4.798981 seconds) notificationworker stdout | 2023-01-17 05:46:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:43 UTC)" (scheduled at 2023-01-17 05:46:43.935629+00:00) notificationworker stdout | 2023-01-17 05:46:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:46:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:46:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:46:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:48.830274+00:00 (in 4.188343 seconds) exportactionlogsworker stdout | 2023-01-17 05:46:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:44 UTC)" (scheduled at 2023-01-17 05:46:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:46:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 46, 44, 642291), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:46:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 44, 642291), True, datetime.datetime(2023, 1, 17, 5, 46, 44, 642291), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:46:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 44, 642291), True, datetime.datetime(2023, 1, 17, 5, 46, 44, 642291), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 46, 44, 642291), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:46:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:46:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:46:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:46:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:59.337413+00:00 (in 11.404440 seconds) namespacegcworker stdout | 2023-01-17 05:46:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:47 UTC)" (scheduled at 2023-01-17 05:46:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:46:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:46:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:58.735066+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 05:46:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:58 UTC)" (scheduled at 2023-01-17 05:46:48.735066+00:00) notificationworker stdout | 2023-01-17 05:46:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 48, 735913), True, datetime.datetime(2023, 1, 17, 5, 46, 48, 735913), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:46:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:46:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:04.738062+00:00 (in 15.999148 seconds) repositorygcworker stdout | 2023-01-17 05:46:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:48 UTC)" (scheduled at 2023-01-17 05:46:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:46:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:46:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:46:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:46:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:24.641816+00:00 (in 35.811107 seconds) exportactionlogsworker stdout | 2023-01-17 05:46:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:48 UTC)" (scheduled at 2023-01-17 05:46:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:46:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:46:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 48, 831082), True, datetime.datetime(2023, 1, 17, 5, 46, 48, 831082), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:46:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:46:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:46:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:46:52,625 [243] [DEBUG] [app] Starting request: urn:request:5dd0cdb4-c833-41a9-a9e3-d81283d8955c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:52,625 [239] [DEBUG] [app] Starting request: urn:request:455f4dc4-a0e7-4e4a-beb3-378c109946db (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:46:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:52,630 [242] [DEBUG] [app] Starting request: urn:request:de479e5b-5b63-4b83-b86c-1a83ffbb3d54 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:52,630 [242] [DEBUG] [app] Ending request: urn:request:de479e5b-5b63-4b83-b86c-1a83ffbb3d54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de479e5b-5b63-4b83-b86c-1a83ffbb3d54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:52,630 [232] [DEBUG] [app] Starting request: urn:request:525c2233-a0a0-4209-b478-f36c075c84fb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:46:52,631 [232] [DEBUG] [app] Ending request: urn:request:525c2233-a0a0-4209-b478-f36c075c84fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:525c2233-a0a0-4209-b478-f36c075c84fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:46:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:52,635 [241] [DEBUG] [app] Starting request: urn:request:129c45aa-29b0-4607-80cc-7bd3687f1405 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:52,636 [244] [DEBUG] [app] Starting request: urn:request:f96c45d7-f9b8-4ed7-a4a6-e31d043864e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:52,636 [241] [DEBUG] [app] Ending request: urn:request:129c45aa-29b0-4607-80cc-7bd3687f1405 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:129c45aa-29b0-4607-80cc-7bd3687f1405', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:52,636 [244] [DEBUG] [app] Ending request: urn:request:f96c45d7-f9b8-4ed7-a4a6-e31d043864e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f96c45d7-f9b8-4ed7-a4a6-e31d043864e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:46:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:46:52,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:52,647 [239] [DEBUG] [app] Ending request: urn:request:455f4dc4-a0e7-4e4a-beb3-378c109946db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:455f4dc4-a0e7-4e4a-beb3-378c109946db', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:52,647 [243] [DEBUG] [app] Ending request: urn:request:5dd0cdb4-c833-41a9-a9e3-d81283d8955c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5dd0cdb4-c833-41a9-a9e3-d81283d8955c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:52,647 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:46:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:46:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:46:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:09.230038+00:00 (in 16.392217 seconds) securityworker stdout | 2023-01-17 05:46:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:22 UTC)" (scheduled at 2023-01-17 05:46:52.837365+00:00) securityworker stdout | 2023-01-17 05:46:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:22 UTC)" executed successfully builder stdout | 2023-01-17 05:46:52,856 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:52,863 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:52,863 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:52,869 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:46:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:46:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:27.226424+00:00 (in 29.999512 seconds) buildlogsarchiver stdout | 2023-01-17 05:46:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:27 UTC)" (scheduled at 2023-01-17 05:46:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:46:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 46, 57, 227344), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:46:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:46:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:46:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:08.735066+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:46:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:08 UTC)" (scheduled at 2023-01-17 05:46:58.735066+00:00) notificationworker stdout | 2023-01-17 05:46:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 58, 735880), True, datetime.datetime(2023, 1, 17, 5, 46, 58, 735880), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:46:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:46:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:47.932490+00:00 (in 48.594604 seconds) namespacegcworker stdout | 2023-01-17 05:46:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:59 UTC)" (scheduled at 2023-01-17 05:46:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:46:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:46:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 59, 338327), True, datetime.datetime(2023, 1, 17, 5, 46, 59, 338327), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:46:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:46:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:46:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:01,291 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:47:01,416 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:47:01,734 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:47:02,534 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:47:02,581 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:47:02,879 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:02,889 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:02,889 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:02,894 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:47:03,854 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:47:04,562 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:47:04,683 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} repositorygcworker stdout | 2023-01-17 05:47:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:47:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:39.030814+00:00 (in 34.292292 seconds) repositorygcworker stdout | 2023-01-17 05:47:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:52:04 UTC)" (scheduled at 2023-01-17 05:47:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:47:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 47, 4, 738893), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:47:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 47, 4, 738893), True, datetime.datetime(2023, 1, 17, 5, 47, 4, 738893), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:47:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 47, 4, 738893), True, datetime.datetime(2023, 1, 17, 5, 47, 4, 738893), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 47, 4, 738893), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:47:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:47:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:52:04 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:47:04,921 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:47:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:47:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:35.848675+00:00 (in 29.999540 seconds) gcworker stdout | 2023-01-17 05:47:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:35 UTC)" (scheduled at 2023-01-17 05:47:05.848675+00:00) gcworker stdout | 2023-01-17 05:47:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724825849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:47:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:47:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:47:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:47:05,981 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:47:05,990 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:47:06,156 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:47:06,159 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:47:06,162 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:47:06,902 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:47:07,625 [244] [DEBUG] [app] Starting request: urn:request:48cc2df0-cf02-47a2-a4e1-81f91d5c5ed9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:07,625 [239] [DEBUG] [app] Starting request: urn:request:6d0579c2-cab5-4d2f-b89b-45b0cdea6015 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:07,630 [242] [DEBUG] [app] Starting request: urn:request:6ac41d9f-2caa-4c34-98d9-41c8fdc78a73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,630 [240] [DEBUG] [app] Starting request: urn:request:3ac970c7-8dd3-492b-a482-b53b7ac5bd55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,630 [242] [DEBUG] [app] Ending request: urn:request:6ac41d9f-2caa-4c34-98d9-41c8fdc78a73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ac41d9f-2caa-4c34-98d9-41c8fdc78a73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:47:07,631 [240] [DEBUG] [app] Ending request: urn:request:3ac970c7-8dd3-492b-a482-b53b7ac5bd55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ac970c7-8dd3-492b-a482-b53b7ac5bd55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:47:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:47:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:07,636 [243] [DEBUG] [app] Starting request: urn:request:7efed6b1-6709-4d3a-9839-7cd6b435f99d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:07,636 [244] [DEBUG] [app] Starting request: urn:request:8aa12346-ca5f-4d45-865b-f203ddbc430a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:07,636 [243] [DEBUG] [app] Ending request: urn:request:7efed6b1-6709-4d3a-9839-7cd6b435f99d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7efed6b1-6709-4d3a-9839-7cd6b435f99d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:07,636 [244] [DEBUG] [app] Ending request: urn:request:8aa12346-ca5f-4d45-865b-f203ddbc430a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8aa12346-ca5f-4d45-865b-f203ddbc430a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:47:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:07,647 [244] [DEBUG] [app] Ending request: urn:request:48cc2df0-cf02-47a2-a4e1-81f91d5c5ed9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48cc2df0-cf02-47a2-a4e1-81f91d5c5ed9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:07,647 [239] [DEBUG] [app] Ending request: urn:request:6d0579c2-cab5-4d2f-b89b-45b0cdea6015 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d0579c2-cab5-4d2f-b89b-45b0cdea6015', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:47:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) logrotateworker stdout | 2023-01-17 05:47:07,664 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:47:07,719 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:47:08,550 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:47:08,554 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:47:08,557 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:47:08,559 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:47:08,564 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:47:08,566 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:47:08,569 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:47:08,572 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:47:08,574 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:47:08,691 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:47:08,693 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:47:08,696 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:47:08,699 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:47:08,702 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:47:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:18.735066+00:00 (in 9.999590 seconds) notificationworker stdout | 2023-01-17 05:47:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:18 UTC)" (scheduled at 2023-01-17 05:47:08.735066+00:00) notificationworker stdout | 2023-01-17 05:47:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 8, 735860), True, datetime.datetime(2023, 1, 17, 5, 47, 8, 735860), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:18 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:47:08,871 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:47:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:22.837365+00:00 (in 13.606890 seconds) securityworker stdout | 2023-01-17 05:47:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:39 UTC)" (scheduled at 2023-01-17 05:47:09.230038+00:00) securityworker stdout | 2023-01-17 05:47:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:47:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:47:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:47:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:47:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:47:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:47:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:09,421 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:47:10,247 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:47:10,258 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:47:10,277 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:47:10,378 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:47:12,905 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:12,914 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:12,914 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:12,920 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:28.735066+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:47:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:28 UTC)" (scheduled at 2023-01-17 05:47:18.735066+00:00) notificationworker stdout | 2023-01-17 05:47:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 18, 735958), True, datetime.datetime(2023, 1, 17, 5, 47, 18, 735958), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:47:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:47:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:25.334138+00:00 (in 5.091482 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:47:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:20 UTC)" (scheduled at 2023-01-17 05:47:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:47:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:47:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 20, 243069), True, datetime.datetime(2023, 1, 17, 5, 47, 20, 243069), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:47:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:47:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:47:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:47:22,625 [239] [DEBUG] [app] Starting request: urn:request:b6ac13c3-934d-4df3-8075-ba7909506181 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:22,625 [243] [DEBUG] [app] Starting request: urn:request:740055f4-7042-4d0b-bffa-5cffb4c3f7db (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:22,630 [242] [DEBUG] [app] Starting request: urn:request:df37483a-ebf2-49ce-bfdf-4204299bec2e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,631 [242] [DEBUG] [app] Ending request: urn:request:df37483a-ebf2-49ce-bfdf-4204299bec2e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df37483a-ebf2-49ce-bfdf-4204299bec2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,630 [236] [DEBUG] [app] Starting request: urn:request:af402541-a3dd-4b5b-8142-4bbaffeb77a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:47:22,631 [236] [DEBUG] [app] Ending request: urn:request:af402541-a3dd-4b5b-8142-4bbaffeb77a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af402541-a3dd-4b5b-8142-4bbaffeb77a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:47:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:22,635 [244] [DEBUG] [app] Starting request: urn:request:a2ba5f2b-1236-408c-8d91-0fc606ce7454 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:22,636 [244] [DEBUG] [app] Ending request: urn:request:a2ba5f2b-1236-408c-8d91-0fc606ce7454 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2ba5f2b-1236-408c-8d91-0fc606ce7454', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:22,636 [241] [DEBUG] [app] Starting request: urn:request:deb1e12c-fd82-494b-bc67-d5048d2d57d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:47:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:22,636 [241] [DEBUG] [app] Ending request: urn:request:deb1e12c-fd82-494b-bc67-d5048d2d57d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:deb1e12c-fd82-494b-bc67-d5048d2d57d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:22,647 [243] [DEBUG] [app] Ending request: urn:request:740055f4-7042-4d0b-bffa-5cffb4c3f7db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:740055f4-7042-4d0b-bffa-5cffb4c3f7db', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:22,647 [239] [DEBUG] [app] Ending request: urn:request:b6ac13c3-934d-4df3-8075-ba7909506181 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6ac13c3-934d-4df3-8075-ba7909506181', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:22,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:47:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:47:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:47:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:39.230038+00:00 (in 16.392220 seconds) securityworker stdout | 2023-01-17 05:47:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:52 UTC)" (scheduled at 2023-01-17 05:47:22.837365+00:00) securityworker stdout | 2023-01-17 05:47:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:52 UTC)" executed successfully builder stdout | 2023-01-17 05:47:22,925 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:22,934 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:22,934 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:22,939 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:47:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:47:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:48.830274+00:00 (in 24.187991 seconds) exportactionlogsworker stdout | 2023-01-17 05:47:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:24 UTC)" (scheduled at 2023-01-17 05:47:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:47:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:47:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:47:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:47:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:20.242198+00:00 (in 54.907586 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:47:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:25 UTC)" (scheduled at 2023-01-17 05:47:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:47:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:47:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:47:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:47:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:57.226424+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2023-01-17 05:47:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:57 UTC)" (scheduled at 2023-01-17 05:47:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:47:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 47, 27, 227245), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:47:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:47:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:47:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:38.735066+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:47:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:38 UTC)" (scheduled at 2023-01-17 05:47:28.735066+00:00) notificationworker stdout | 2023-01-17 05:47:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 28, 735976), True, datetime.datetime(2023, 1, 17, 5, 47, 28, 735976), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:31,325 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:47:31,433 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:47:31,739 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:47:32,570 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:47:32,616 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:47:32,944 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:32,953 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:32,953 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:32,959 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:47:33,889 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:47:34,568 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:47:34,718 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:47:34,950 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:47:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:47:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:05.848675+00:00 (in 29.999554 seconds) gcworker stdout | 2023-01-17 05:47:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:05 UTC)" (scheduled at 2023-01-17 05:47:35.848675+00:00) gcworker stdout | 2023-01-17 05:47:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724855849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:47:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:47:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:47:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:05 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:47:36,008 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:47:36,016 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:47:36,162 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:47:36,166 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:47:36,168 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:47:36,934 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:47:37,626 [239] [DEBUG] [app] Starting request: urn:request:a0df2bb1-e242-49d5-b9d2-4dff876b7508 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:37,626 [241] [DEBUG] [app] Starting request: urn:request:bf6d217a-ba9d-4768-9a67-70df63dfd397 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:37,631 [232] [DEBUG] [app] Starting request: urn:request:5e182c9a-06d7-44fc-83b2-c6bdc8e21e80 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:37,631 [242] [DEBUG] [app] Starting request: urn:request:d1b7019d-cd0e-4381-afd9-c80cbcf4440e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:37,631 [232] [DEBUG] [app] Ending request: urn:request:5e182c9a-06d7-44fc-83b2-c6bdc8e21e80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e182c9a-06d7-44fc-83b2-c6bdc8e21e80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:47:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:47:37,632 [242] [DEBUG] [app] Ending request: urn:request:d1b7019d-cd0e-4381-afd9-c80cbcf4440e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d1b7019d-cd0e-4381-afd9-c80cbcf4440e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:47:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:47:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:37,636 [244] [DEBUG] [app] Starting request: urn:request:8c315403-bd04-4ceb-b476-80b3355ca3b4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:37,637 [241] [DEBUG] [app] Starting request: urn:request:7bc4a455-3291-472e-a8ca-b46d5ba460db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:37,637 [244] [DEBUG] [app] Ending request: urn:request:8c315403-bd04-4ceb-b476-80b3355ca3b4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c315403-bd04-4ceb-b476-80b3355ca3b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:37,637 [241] [DEBUG] [app] Ending request: urn:request:7bc4a455-3291-472e-a8ca-b46d5ba460db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7bc4a455-3291-472e-a8ca-b46d5ba460db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:47:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:37,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:37,648 [239] [DEBUG] [app] Ending request: urn:request:a0df2bb1-e242-49d5-b9d2-4dff876b7508 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0df2bb1-e242-49d5-b9d2-4dff876b7508', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:37,648 [241] [DEBUG] [app] Ending request: urn:request:bf6d217a-ba9d-4768-9a67-70df63dfd397 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf6d217a-ba9d-4768-9a67-70df63dfd397', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:47:37,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:47:37,685 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:47:37,736 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:47:38,560 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:47:38,562 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:47:38,565 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:47:38,567 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:47:38,570 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:47:38,574 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:47:38,577 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:47:38,580 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:47:38,582 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:47:38,697 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:47:38,700 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:47:38,702 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:47:38,705 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:47:38,708 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:47:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:43.935629+00:00 (in 5.200171 seconds) notificationworker stdout | 2023-01-17 05:47:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:48 UTC)" (scheduled at 2023-01-17 05:47:38.735066+00:00) notificationworker stdout | 2023-01-17 05:47:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 38, 735824), True, datetime.datetime(2023, 1, 17, 5, 47, 38, 735824), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:48 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:47:38,907 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:47:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:47:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:48.738458+00:00 (in 9.707207 seconds) repositorygcworker stdout | 2023-01-17 05:47:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:39 UTC)" (scheduled at 2023-01-17 05:47:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:47:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:47:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 39, 31681), True, datetime.datetime(2023, 1, 17, 5, 47, 39, 31681), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:47:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:47:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:47:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:52.837365+00:00 (in 13.606867 seconds) securityworker stdout | 2023-01-17 05:47:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:09 UTC)" (scheduled at 2023-01-17 05:47:39.230038+00:00) securityworker stdout | 2023-01-17 05:47:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:47:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:47:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:47:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:39,456 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:47:40,270 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:47:40,291 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:47:40,311 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:47:40,413 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:47:42,966 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:42,974 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:42,974 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:42,980 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:48.735066+00:00 (in 4.798998 seconds) notificationworker stdout | 2023-01-17 05:47:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:43 UTC)" (scheduled at 2023-01-17 05:47:43.935629+00:00) notificationworker stdout | 2023-01-17 05:47:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:47:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:47:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:47:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:59.337413+00:00 (in 11.404461 seconds) namespacegcworker stdout | 2023-01-17 05:47:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:47 UTC)" (scheduled at 2023-01-17 05:47:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:47:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:47:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:58.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 05:47:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:58 UTC)" (scheduled at 2023-01-17 05:47:48.735066+00:00) notificationworker stdout | 2023-01-17 05:47:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 48, 735943), True, datetime.datetime(2023, 1, 17, 5, 47, 48, 735943), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:47:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:47:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:39.030814+00:00 (in 50.291930 seconds) repositorygcworker stdout | 2023-01-17 05:47:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:48 UTC)" (scheduled at 2023-01-17 05:47:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:47:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:47:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:47:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:47:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:24.641816+00:00 (in 35.811100 seconds) exportactionlogsworker stdout | 2023-01-17 05:47:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:48 UTC)" (scheduled at 2023-01-17 05:47:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:47:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:47:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 48, 831124), True, datetime.datetime(2023, 1, 17, 5, 47, 48, 831124), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:47:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:47:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:47:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:47:52,625 [244] [DEBUG] [app] Starting request: urn:request:084ce7f6-5994-4119-b0d7-2c3cedfdadfe (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:52,625 [239] [DEBUG] [app] Starting request: urn:request:99d7e043-5761-43e6-a07c-991ce6b9ad1b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:47:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:52,630 [240] [DEBUG] [app] Starting request: urn:request:b27ba7c9-40dc-445c-9f0e-b1e3a4c56310 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:52,630 [235] [DEBUG] [app] Starting request: urn:request:2974d750-319a-45e1-adff-97f6ec38f983 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:52,630 [240] [DEBUG] [app] Ending request: urn:request:b27ba7c9-40dc-445c-9f0e-b1e3a4c56310 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b27ba7c9-40dc-445c-9f0e-b1e3a4c56310', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:47:52,631 [235] [DEBUG] [app] Ending request: urn:request:2974d750-319a-45e1-adff-97f6ec38f983 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2974d750-319a-45e1-adff-97f6ec38f983', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:47:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:47:52,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:47:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:52,635 [243] [DEBUG] [app] Starting request: urn:request:8a579812-5065-4f8f-a9e1-10971790b939 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:52,635 [243] [DEBUG] [app] Ending request: urn:request:8a579812-5065-4f8f-a9e1-10971790b939 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8a579812-5065-4f8f-a9e1-10971790b939', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:52,635 [241] [DEBUG] [app] Starting request: urn:request:353a9cb8-3ce0-44e8-8398-31cf8edbc01a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:52,636 [241] [DEBUG] [app] Ending request: urn:request:353a9cb8-3ce0-44e8-8398-31cf8edbc01a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:353a9cb8-3ce0-44e8-8398-31cf8edbc01a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:47:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:52,646 [244] [DEBUG] [app] Ending request: urn:request:084ce7f6-5994-4119-b0d7-2c3cedfdadfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:084ce7f6-5994-4119-b0d7-2c3cedfdadfe', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:52,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:52,646 [239] [DEBUG] [app] Ending request: urn:request:99d7e043-5761-43e6-a07c-991ce6b9ad1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99d7e043-5761-43e6-a07c-991ce6b9ad1b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:47:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:47:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:09.230038+00:00 (in 16.392261 seconds) securityworker stdout | 2023-01-17 05:47:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:22 UTC)" (scheduled at 2023-01-17 05:47:52.837365+00:00) securityworker stdout | 2023-01-17 05:47:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:22 UTC)" executed successfully builder stdout | 2023-01-17 05:47:52,990 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:52,998 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:52,998 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:53,004 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:47:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:47:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:27.226424+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 05:47:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:27 UTC)" (scheduled at 2023-01-17 05:47:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:47:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 47, 57, 227272), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:47:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:47:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:47:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:08.735066+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:47:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:08 UTC)" (scheduled at 2023-01-17 05:47:58.735066+00:00) notificationworker stdout | 2023-01-17 05:47:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 58, 735980), True, datetime.datetime(2023, 1, 17, 5, 47, 58, 735980), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:58,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:58,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:47:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:47:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:47.932490+00:00 (in 48.594632 seconds) namespacegcworker stdout | 2023-01-17 05:47:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:59 UTC)" (scheduled at 2023-01-17 05:47:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:47:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:47:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 59, 338289), True, datetime.datetime(2023, 1, 17, 5, 47, 59, 338289), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:47:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:47:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:47:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:01,351 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:48:01,469 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:48:01,774 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:48:02,586 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:48:02,650 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:48:03,014 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:03,023 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:03,023 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:03,029 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:48:03,924 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:48:04,583 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:48:04,753 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:48:04,961 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:48:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:48:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:35.848675+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 05:48:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:35 UTC)" (scheduled at 2023-01-17 05:48:05.848675+00:00) gcworker stdout | 2023-01-17 05:48:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724885849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:48:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:48:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:48:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:35 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:48:06,030 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:48:06,051 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:48:06,168 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:48:06,172 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:48:06,174 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:48:06,970 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:48:07,626 [239] [DEBUG] [app] Starting request: urn:request:6239b332-e395-4fc8-a546-aceffccbed8d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:07,626 [244] [DEBUG] [app] Starting request: urn:request:d233087f-6d6a-4607-92c1-00bca4d1b925 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:07,631 [240] [DEBUG] [app] Starting request: urn:request:061925a5-da0e-450a-a325-f557c1ce534f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:07,631 [242] [DEBUG] [app] Starting request: urn:request:18900743-e141-4d36-90ff-30c9e8424355 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:07,631 [240] [DEBUG] [app] Ending request: urn:request:061925a5-da0e-450a-a325-f557c1ce534f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:061925a5-da0e-450a-a325-f557c1ce534f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:48:07,632 [242] [DEBUG] [app] Ending request: urn:request:18900743-e141-4d36-90ff-30c9e8424355 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18900743-e141-4d36-90ff-30c9e8424355', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:48:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:48:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:07,636 [241] [DEBUG] [app] Starting request: urn:request:8b1dd361-6423-457d-98e9-6470e2709e24 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:07,637 [239] [DEBUG] [app] Starting request: urn:request:a1e3cbf7-b8bd-4e19-9111-9ded472b3c87 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:07,637 [241] [DEBUG] [app] Ending request: urn:request:8b1dd361-6423-457d-98e9-6470e2709e24 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b1dd361-6423-457d-98e9-6470e2709e24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:07,637 [239] [DEBUG] [app] Ending request: urn:request:a1e3cbf7-b8bd-4e19-9111-9ded472b3c87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1e3cbf7-b8bd-4e19-9111-9ded472b3c87', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:07,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:07,648 [244] [DEBUG] [app] Ending request: urn:request:d233087f-6d6a-4607-92c1-00bca4d1b925 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d233087f-6d6a-4607-92c1-00bca4d1b925', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:07,648 [239] [DEBUG] [app] Ending request: urn:request:6239b332-e395-4fc8-a546-aceffccbed8d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6239b332-e395-4fc8-a546-aceffccbed8d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:07,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:48:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:48:07,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:48:07,718 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:48:07,758 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:48:08,569 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:48:08,572 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:48:08,574 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:48:08,577 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:48:08,579 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:48:08,583 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:48:08,586 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:48:08,588 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:48:08,593 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:48:08,703 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:48:08,706 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:48:08,708 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:48:08,712 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:48:08,715 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:48:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:18.735066+00:00 (in 9.999552 seconds) notificationworker stdout | 2023-01-17 05:48:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:18 UTC)" (scheduled at 2023-01-17 05:48:08.735066+00:00) notificationworker stdout | 2023-01-17 05:48:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 8, 735926), True, datetime.datetime(2023, 1, 17, 5, 48, 8, 735926), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:18 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:48:08,933 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:48:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:22.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 05:48:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:39 UTC)" (scheduled at 2023-01-17 05:48:09.230038+00:00) securityworker stdout | 2023-01-17 05:48:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:48:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:48:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:09,232 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:48:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:48:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:48:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:09,243 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:09,243 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:09,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:48:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:09,492 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:48:10,295 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:48:10,303 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:48:10,346 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:48:10,447 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:48:13,039 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:13,048 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:13,048 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:13,053 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:28.735066+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:48:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:28 UTC)" (scheduled at 2023-01-17 05:48:18.735066+00:00) notificationworker stdout | 2023-01-17 05:48:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 18, 735958), True, datetime.datetime(2023, 1, 17, 5, 48, 18, 735958), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:48:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:48:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:25.334138+00:00 (in 5.091465 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:48:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:20 UTC)" (scheduled at 2023-01-17 05:48:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:48:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:48:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 20, 243093), True, datetime.datetime(2023, 1, 17, 5, 48, 20, 243093), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:48:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:48:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:48:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:48:22,625 [243] [DEBUG] [app] Starting request: urn:request:71f59035-3d9d-472c-b0cd-9aa5c5c32f4d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:22,625 [239] [DEBUG] [app] Starting request: urn:request:a4ec8a5e-e0be-42ce-a8b8-6fcc0c8762db (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:22,630 [242] [DEBUG] [app] Starting request: urn:request:8dd10965-352e-44f1-a40c-03d90174dd6b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:22,631 [242] [DEBUG] [app] Ending request: urn:request:8dd10965-352e-44f1-a40c-03d90174dd6b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8dd10965-352e-44f1-a40c-03d90174dd6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:22,631 [232] [DEBUG] [app] Starting request: urn:request:1dda8e74-2b97-4d9f-925d-a8ced5f1eb6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:48:22,631 [232] [DEBUG] [app] Ending request: urn:request:1dda8e74-2b97-4d9f-925d-a8ced5f1eb6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1dda8e74-2b97-4d9f-925d-a8ced5f1eb6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:48:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:22,635 [244] [DEBUG] [app] Starting request: urn:request:e0685943-8e97-4078-ad14-e46b25e2115c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:22,636 [244] [DEBUG] [app] Ending request: urn:request:e0685943-8e97-4078-ad14-e46b25e2115c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0685943-8e97-4078-ad14-e46b25e2115c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:22,636 [241] [DEBUG] [app] Starting request: urn:request:256bea4c-64ce-46b9-846e-2b61e79aa453 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:48:22,636 [241] [DEBUG] [app] Ending request: urn:request:256bea4c-64ce-46b9-846e-2b61e79aa453 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:256bea4c-64ce-46b9-846e-2b61e79aa453', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:22,647 [243] [DEBUG] [app] Ending request: urn:request:71f59035-3d9d-472c-b0cd-9aa5c5c32f4d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71f59035-3d9d-472c-b0cd-9aa5c5c32f4d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:22,647 [239] [DEBUG] [app] Ending request: urn:request:a4ec8a5e-e0be-42ce-a8b8-6fcc0c8762db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a4ec8a5e-e0be-42ce-a8b8-6fcc0c8762db', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:48:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:48:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:39.230038+00:00 (in 16.392204 seconds) securityworker stdout | 2023-01-17 05:48:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:52 UTC)" (scheduled at 2023-01-17 05:48:22.837365+00:00) securityworker stdout | 2023-01-17 05:48:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:52 UTC)" executed successfully builder stdout | 2023-01-17 05:48:23,064 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:23,072 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:23,072 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:23,078 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:48:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:48:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:48.830274+00:00 (in 24.188030 seconds) exportactionlogsworker stdout | 2023-01-17 05:48:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:24 UTC)" (scheduled at 2023-01-17 05:48:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:48:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:48:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:48:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:48:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:20.242198+00:00 (in 54.907610 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:48:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:25 UTC)" (scheduled at 2023-01-17 05:48:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:48:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:48:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:48:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:48:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:57.226424+00:00 (in 29.999499 seconds) buildlogsarchiver stdout | 2023-01-17 05:48:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:57 UTC)" (scheduled at 2023-01-17 05:48:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:48:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 48, 27, 227295), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:48:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:48:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:48:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:38.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 05:48:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:38 UTC)" (scheduled at 2023-01-17 05:48:28.735066+00:00) notificationworker stdout | 2023-01-17 05:48:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 28, 735983), True, datetime.datetime(2023, 1, 17, 5, 48, 28, 735983), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:31,366 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:48:31,504 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:48:31,810 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:48:32,603 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:48:32,685 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:48:33,088 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:33,097 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:33,097 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:33,103 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:48:33,931 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:48:34,611 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:48:34,776 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:48:34,996 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:48:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:48:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:05.848675+00:00 (in 29.999525 seconds) gcworker stdout | 2023-01-17 05:48:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:05 UTC)" (scheduled at 2023-01-17 05:48:35.848675+00:00) gcworker stdout | 2023-01-17 05:48:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724915849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:48:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:48:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:48:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:05 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:48:36,053 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:48:36,086 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:48:36,174 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:48:36,178 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:48:36,180 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:48:37,006 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:48:37,625 [239] [DEBUG] [app] Starting request: urn:request:eb7a7334-edca-4160-ab1b-39cc9885aba1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:37,625 [244] [DEBUG] [app] Starting request: urn:request:7c6d688a-b475-42e9-a4df-f057c7d037a8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:37,630 [232] [DEBUG] [app] Starting request: urn:request:855be5e1-459f-4c66-b89a-c832746b9ca2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,630 [232] [DEBUG] [app] Ending request: urn:request:855be5e1-459f-4c66-b89a-c832746b9ca2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:855be5e1-459f-4c66-b89a-c832746b9ca2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:37,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:48:37,631 [242] [DEBUG] [app] Starting request: urn:request:fb4a2647-6df6-4584-8103-81f4295bb4c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,632 [242] [DEBUG] [app] Ending request: urn:request:fb4a2647-6df6-4584-8103-81f4295bb4c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fb4a2647-6df6-4584-8103-81f4295bb4c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:48:37,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:37,635 [243] [DEBUG] [app] Starting request: urn:request:0d9ae610-f4b6-4ada-94cc-29e71c02410b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:37,635 [243] [DEBUG] [app] Ending request: urn:request:0d9ae610-f4b6-4ada-94cc-29e71c02410b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d9ae610-f4b6-4ada-94cc-29e71c02410b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:37,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:37,637 [243] [DEBUG] [app] Starting request: urn:request:272cddf9-b6d0-46f0-9f25-3b46c5aacf6a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:37,637 [243] [DEBUG] [app] Ending request: urn:request:272cddf9-b6d0-46f0-9f25-3b46c5aacf6a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:272cddf9-b6d0-46f0-9f25-3b46c5aacf6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:37,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:37,647 [239] [DEBUG] [app] Ending request: urn:request:eb7a7334-edca-4160-ab1b-39cc9885aba1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb7a7334-edca-4160-ab1b-39cc9885aba1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:37,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:48:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:48:37,648 [244] [DEBUG] [app] Ending request: urn:request:7c6d688a-b475-42e9-a4df-f057c7d037a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c6d688a-b475-42e9-a4df-f057c7d037a8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) logrotateworker stdout | 2023-01-17 05:48:37,729 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:48:37,793 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:48:38,578 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:48:38,581 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:48:38,583 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:48:38,586 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:48:38,589 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:48:38,591 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:48:38,593 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:48:38,599 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:48:38,602 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:48:38,709 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:48:38,711 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:48:38,716 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:48:38,719 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:48:38,721 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:48:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:43.935629+00:00 (in 5.200156 seconds) notificationworker stdout | 2023-01-17 05:48:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:48 UTC)" (scheduled at 2023-01-17 05:48:38.735066+00:00) notificationworker stdout | 2023-01-17 05:48:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 38, 735877), True, datetime.datetime(2023, 1, 17, 5, 48, 38, 735877), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:38,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:38,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:48 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:48:38,956 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:48:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:48:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:48.738458+00:00 (in 9.707216 seconds) repositorygcworker stdout | 2023-01-17 05:48:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:39 UTC)" (scheduled at 2023-01-17 05:48:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:48:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:48:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 39, 31724), True, datetime.datetime(2023, 1, 17, 5, 48, 39, 31724), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:48:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:48:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:48:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:52.837365+00:00 (in 13.606889 seconds) securityworker stdout | 2023-01-17 05:48:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:09 UTC)" (scheduled at 2023-01-17 05:48:39.230038+00:00) securityworker stdout | 2023-01-17 05:48:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:48:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:48:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:48:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:39,528 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:48:40,330 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:48:40,333 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:48:40,351 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:48:40,458 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:48:43,113 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:43,127 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:43,127 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:43,133 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:48.735066+00:00 (in 4.798891 seconds) notificationworker stdout | 2023-01-17 05:48:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:43 UTC)" (scheduled at 2023-01-17 05:48:43.935629+00:00) notificationworker stdout | 2023-01-17 05:48:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:48:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:48:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:48:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:59.337413+00:00 (in 11.404423 seconds) namespacegcworker stdout | 2023-01-17 05:48:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:47 UTC)" (scheduled at 2023-01-17 05:48:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:48:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:48:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:56.935055+00:00 (in 8.199435 seconds) notificationworker stdout | 2023-01-17 05:48:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:58 UTC)" (scheduled at 2023-01-17 05:48:48.735066+00:00) notificationworker stdout | 2023-01-17 05:48:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 48, 736077), True, datetime.datetime(2023, 1, 17, 5, 48, 48, 736077), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:48:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:48:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:39.030814+00:00 (in 50.291848 seconds) repositorygcworker stdout | 2023-01-17 05:48:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:48 UTC)" (scheduled at 2023-01-17 05:48:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:48:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:48:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:48:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:48:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:24.641816+00:00 (in 35.811084 seconds) exportactionlogsworker stdout | 2023-01-17 05:48:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:48 UTC)" (scheduled at 2023-01-17 05:48:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:48:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:48:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 48, 831141), True, datetime.datetime(2023, 1, 17, 5, 48, 48, 831141), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:48:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:48:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:48:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:48:52,626 [239] [DEBUG] [app] Starting request: urn:request:15cdf248-10d1-456e-ba39-7a71ad852e9e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:52,626 [244] [DEBUG] [app] Starting request: urn:request:d8ad758c-30c7-4295-8ca8-05686515c73d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:48:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:52,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:52,631 [242] [DEBUG] [app] Starting request: urn:request:6f2bd77c-b912-4ae2-9a12-c2a9b96d3e3c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,632 [242] [DEBUG] [app] Ending request: urn:request:6f2bd77c-b912-4ae2-9a12-c2a9b96d3e3c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f2bd77c-b912-4ae2-9a12-c2a9b96d3e3c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,632 [240] [DEBUG] [app] Starting request: urn:request:2410a3bd-06da-4573-9966-699cac9eea8f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:48:52,632 [240] [DEBUG] [app] Ending request: urn:request:2410a3bd-06da-4573-9966-699cac9eea8f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2410a3bd-06da-4573-9966-699cac9eea8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:48:52,633 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:48:52,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:52,636 [239] [DEBUG] [app] Starting request: urn:request:4eccf1c9-f46f-4ccd-a571-7619f971d232 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:52,636 [239] [DEBUG] [app] Ending request: urn:request:4eccf1c9-f46f-4ccd-a571-7619f971d232 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4eccf1c9-f46f-4ccd-a571-7619f971d232', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:52,637 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:52,638 [243] [DEBUG] [app] Starting request: urn:request:5330e0c6-be8f-489f-bf53-321304488740 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:52,639 [243] [DEBUG] [app] Ending request: urn:request:5330e0c6-be8f-489f-bf53-321304488740 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5330e0c6-be8f-489f-bf53-321304488740', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:52,639 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:48:52,639 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:52,640 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:52,644 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:52,647 [244] [DEBUG] [app] Ending request: urn:request:d8ad758c-30c7-4295-8ca8-05686515c73d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8ad758c-30c7-4295-8ca8-05686515c73d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:48:52,649 [239] [DEBUG] [app] Ending request: urn:request:15cdf248-10d1-456e-ba39-7a71ad852e9e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15cdf248-10d1-456e-ba39-7a71ad852e9e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:52,650 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) securityworker stdout | 2023-01-17 05:48:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:09.230038+00:00 (in 16.392211 seconds) securityworker stdout | 2023-01-17 05:48:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:22 UTC)" (scheduled at 2023-01-17 05:48:52.837365+00:00) securityworker stdout | 2023-01-17 05:48:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:52,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:22 UTC)" executed successfully builder stdout | 2023-01-17 05:48:53,143 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:53,152 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:53,152 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:53,157 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:58.735066+00:00 (in 1.799553 seconds) notificationworker stdout | 2023-01-17 05:48:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:53:56 UTC)" (scheduled at 2023-01-17 05:48:56.935055+00:00) notificationworker stdout | 2023-01-17 05:48:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 48, 56, 935895), 'notification/%']) notificationworker stdout | 2023-01-17 05:48:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 48, 56, 935895), True, datetime.datetime(2023, 1, 17, 5, 48, 56, 935895), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:48:56,946 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 48, 56, 935895), True, datetime.datetime(2023, 1, 17, 5, 48, 56, 935895), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 48, 56, 935895), 'notification/%']) notificationworker stdout | 2023-01-17 05:48:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:53:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:48:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:48:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:27.226424+00:00 (in 29.999555 seconds) buildlogsarchiver stdout | 2023-01-17 05:48:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:27 UTC)" (scheduled at 2023-01-17 05:48:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:48:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 48, 57, 227235), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:48:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:48:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:48:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:08.735066+00:00 (in 9.999467 seconds) notificationworker stdout | 2023-01-17 05:48:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:08 UTC)" (scheduled at 2023-01-17 05:48:58.735066+00:00) notificationworker stdout | 2023-01-17 05:48:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 58, 736097), True, datetime.datetime(2023, 1, 17, 5, 48, 58, 736097), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:48:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:48:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:15.931994+00:00 (in 16.594143 seconds) namespacegcworker stdout | 2023-01-17 05:48:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:59 UTC)" (scheduled at 2023-01-17 05:48:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:48:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:48:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 59, 338258), True, datetime.datetime(2023, 1, 17, 5, 48, 59, 338258), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:48:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:48:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:48:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:01,384 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:49:01,513 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:49:01,845 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:49:02,613 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:49:02,720 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:49:03,168 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:03,176 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:03,176 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:03,182 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:49:03,951 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:49:04,647 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:49:04,785 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:49:05,031 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:49:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:49:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:35.848675+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 05:49:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:35 UTC)" (scheduled at 2023-01-17 05:49:05.848675+00:00) gcworker stdout | 2023-01-17 05:49:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724945849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:49:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:49:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:49:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:35 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:49:06,077 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:49:06,092 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:49:06,181 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:49:06,185 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:49:06,187 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:49:07,025 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:49:07,625 [239] [DEBUG] [app] Starting request: urn:request:f46e6bc7-0fdd-4a21-bfb8-558da2f8f4ed (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:07,625 [244] [DEBUG] [app] Starting request: urn:request:93e5d700-68aa-4b3b-8af6-75396ab7256c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:07,630 [242] [DEBUG] [app] Starting request: urn:request:909ffc49-022b-47fb-9f3b-4fb29930baa3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:07,631 [242] [DEBUG] [app] Ending request: urn:request:909ffc49-022b-47fb-9f3b-4fb29930baa3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:909ffc49-022b-47fb-9f3b-4fb29930baa3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:07,631 [237] [DEBUG] [app] Starting request: urn:request:23ee30b3-663a-4c8c-ae79-bcf4742efe27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:49:07,631 [237] [DEBUG] [app] Ending request: urn:request:23ee30b3-663a-4c8c-ae79-bcf4742efe27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23ee30b3-663a-4c8c-ae79-bcf4742efe27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:49:07,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:07,635 [243] [DEBUG] [app] Starting request: urn:request:e7595dd0-e27f-4f3f-8fb1-500f576778c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:07,635 [239] [DEBUG] [app] Starting request: urn:request:00367506-62bb-464c-bdef-53cc9cb76525 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:07,636 [243] [DEBUG] [app] Ending request: urn:request:e7595dd0-e27f-4f3f-8fb1-500f576778c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7595dd0-e27f-4f3f-8fb1-500f576778c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:07,636 [239] [DEBUG] [app] Ending request: urn:request:00367506-62bb-464c-bdef-53cc9cb76525 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00367506-62bb-464c-bdef-53cc9cb76525', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:49:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:07,646 [239] [DEBUG] [app] Ending request: urn:request:f46e6bc7-0fdd-4a21-bfb8-558da2f8f4ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f46e6bc7-0fdd-4a21-bfb8-558da2f8f4ed', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:07,647 [244] [DEBUG] [app] Ending request: urn:request:93e5d700-68aa-4b3b-8af6-75396ab7256c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93e5d700-68aa-4b3b-8af6-75396ab7256c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:49:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:49:07,758 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:49:07,804 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:49:08,586 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:49:08,589 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:49:08,593 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:49:08,597 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:49:08,600 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:49:08,602 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:49:08,604 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:49:08,609 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:49:08,611 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:49:08,716 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:49:08,718 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:49:08,721 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:49:08,724 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:49:08,727 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:49:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:18.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 05:49:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:18 UTC)" (scheduled at 2023-01-17 05:49:08.735066+00:00) notificationworker stdout | 2023-01-17 05:49:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 8, 735921), True, datetime.datetime(2023, 1, 17, 5, 49, 8, 735921), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:18 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:49:08,966 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:49:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:22.837365+00:00 (in 13.606870 seconds) securityworker stdout | 2023-01-17 05:49:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:39 UTC)" (scheduled at 2023-01-17 05:49:09.230038+00:00) securityworker stdout | 2023-01-17 05:49:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:49:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:49:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:49:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:49:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:49:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:49:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:09,563 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:49:10,353 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:49:10,356 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:49:10,358 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:49:10,490 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:49:13,192 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:13,201 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:13,201 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:13,207 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:49:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:49:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:47.932490+00:00 (in 32.000028 seconds) namespacegcworker stdout | 2023-01-17 05:49:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:54:15 UTC)" (scheduled at 2023-01-17 05:49:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:49:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 49, 15, 932838), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:49:15,939 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 49, 15, 932838), True, datetime.datetime(2023, 1, 17, 5, 49, 15, 932838), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:49:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 49, 15, 932838), True, datetime.datetime(2023, 1, 17, 5, 49, 15, 932838), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 49, 15, 932838), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:49:15,943 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:49:15,943 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:54:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:28.735066+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 05:49:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:28 UTC)" (scheduled at 2023-01-17 05:49:18.735066+00:00) notificationworker stdout | 2023-01-17 05:49:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 18, 736004), True, datetime.datetime(2023, 1, 17, 5, 49, 18, 736004), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:49:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:49:20,243 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:25.334138+00:00 (in 5.091271 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:49:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:20 UTC)" (scheduled at 2023-01-17 05:49:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:49:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:49:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 20, 243652), True, datetime.datetime(2023, 1, 17, 5, 49, 20, 243652), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:49:20,253 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:49:20,253 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:49:20,253 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:49:22,625 [239] [DEBUG] [app] Starting request: urn:request:9568edd3-42c6-49be-bd10-a1c356a854ce (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:22,625 [243] [DEBUG] [app] Starting request: urn:request:cf341796-5df3-4fa1-947e-0531fbe13cdb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:22,630 [232] [DEBUG] [app] Starting request: urn:request:77d930c0-ef98-456d-8bdc-3b9305c88a4f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:22,631 [242] [DEBUG] [app] Starting request: urn:request:27471412-8c56-41f4-907c-11737d8b00fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:22,631 [232] [DEBUG] [app] Ending request: urn:request:77d930c0-ef98-456d-8bdc-3b9305c88a4f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77d930c0-ef98-456d-8bdc-3b9305c88a4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:49:22,631 [242] [DEBUG] [app] Ending request: urn:request:27471412-8c56-41f4-907c-11737d8b00fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27471412-8c56-41f4-907c-11737d8b00fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:49:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:22,636 [244] [DEBUG] [app] Starting request: urn:request:baf8ac91-613b-4dc6-b021-5506934a242c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:22,636 [239] [DEBUG] [app] Starting request: urn:request:a21665e3-2e1e-4010-93b7-ef924c041e48 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:22,636 [239] [DEBUG] [app] Ending request: urn:request:a21665e3-2e1e-4010-93b7-ef924c041e48 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a21665e3-2e1e-4010-93b7-ef924c041e48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:22,636 [244] [DEBUG] [app] Ending request: urn:request:baf8ac91-613b-4dc6-b021-5506934a242c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:baf8ac91-613b-4dc6-b021-5506934a242c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:22,647 [243] [DEBUG] [app] Ending request: urn:request:cf341796-5df3-4fa1-947e-0531fbe13cdb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf341796-5df3-4fa1-947e-0531fbe13cdb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:22,647 [239] [DEBUG] [app] Ending request: urn:request:9568edd3-42c6-49be-bd10-a1c356a854ce (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9568edd3-42c6-49be-bd10-a1c356a854ce', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:49:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:49:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:39.230038+00:00 (in 16.392203 seconds) securityworker stdout | 2023-01-17 05:49:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:52 UTC)" (scheduled at 2023-01-17 05:49:22.837365+00:00) securityworker stdout | 2023-01-17 05:49:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:22,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:22,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:22,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:52 UTC)" executed successfully builder stdout | 2023-01-17 05:49:23,216 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:23,224 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:23,224 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:23,230 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:49:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:49:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:48.830274+00:00 (in 24.187984 seconds) exportactionlogsworker stdout | 2023-01-17 05:49:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:24 UTC)" (scheduled at 2023-01-17 05:49:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:49:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:49:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:49:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:49:25,335 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:20.242198+00:00 (in 54.907197 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:49:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:25 UTC)" (scheduled at 2023-01-17 05:49:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:49:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:49:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:49:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:49:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:57.226424+00:00 (in 29.999582 seconds) buildlogsarchiver stdout | 2023-01-17 05:49:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:57 UTC)" (scheduled at 2023-01-17 05:49:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:49:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 49, 27, 227187), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:49:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:49:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:49:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:38.735066+00:00 (in 9.999419 seconds) notificationworker stdout | 2023-01-17 05:49:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:38 UTC)" (scheduled at 2023-01-17 05:49:28.735066+00:00) notificationworker stdout | 2023-01-17 05:49:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 28, 736141), True, datetime.datetime(2023, 1, 17, 5, 49, 28, 736141), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:31,419 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:49:31,548 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:49:31,881 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:49:32,638 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:49:32,756 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:49:33,240 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:33,249 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:33,249 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:33,255 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:49:33,957 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:49:34,675 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:49:34,807 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:49:35,061 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:49:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:49:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:05.848675+00:00 (in 29.999495 seconds) gcworker stdout | 2023-01-17 05:49:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:05 UTC)" (scheduled at 2023-01-17 05:49:35.848675+00:00) gcworker stdout | 2023-01-17 05:49:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724975849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:49:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:49:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:49:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:05 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:49:36,112 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:49:36,127 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:49:36,187 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:49:36,192 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:49:36,194 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:49:37,036 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:49:37,626 [241] [DEBUG] [app] Starting request: urn:request:0c296e90-c329-4988-85be-fd353eebe620 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:37,626 [239] [DEBUG] [app] Starting request: urn:request:97ab2d72-5fb9-41d8-8df7-d0de0be351e8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:37,630 [232] [DEBUG] [app] Starting request: urn:request:cf0e612a-69ca-4ed8-9920-f3597e86450d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,631 [232] [DEBUG] [app] Ending request: urn:request:cf0e612a-69ca-4ed8-9920-f3597e86450d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf0e612a-69ca-4ed8-9920-f3597e86450d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,631 [242] [DEBUG] [app] Starting request: urn:request:ab7e8777-892d-4b88-b4ff-e3c0d4a81951 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:49:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:49:37,632 [242] [DEBUG] [app] Ending request: urn:request:ab7e8777-892d-4b88-b4ff-e3c0d4a81951 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab7e8777-892d-4b88-b4ff-e3c0d4a81951', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:49:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:37,636 [243] [DEBUG] [app] Starting request: urn:request:f44e3f39-bf12-4e2f-a3b9-eb884c2d9a26 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:37,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:37,636 [243] [DEBUG] [app] Ending request: urn:request:f44e3f39-bf12-4e2f-a3b9-eb884c2d9a26 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f44e3f39-bf12-4e2f-a3b9-eb884c2d9a26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:37,637 [244] [DEBUG] [app] Starting request: urn:request:50cbdc9f-e780-4e37-8a4d-325cb0645297 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:37,637 [244] [DEBUG] [app] Ending request: urn:request:50cbdc9f-e780-4e37-8a4d-325cb0645297 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50cbdc9f-e780-4e37-8a4d-325cb0645297', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:37,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:37,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:37,647 [239] [DEBUG] [app] Ending request: urn:request:97ab2d72-5fb9-41d8-8df7-d0de0be351e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97ab2d72-5fb9-41d8-8df7-d0de0be351e8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:49:37,648 [241] [DEBUG] [app] Ending request: urn:request:0c296e90-c329-4988-85be-fd353eebe620 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c296e90-c329-4988-85be-fd353eebe620', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:37,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:49:37,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:49:37,794 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:49:37,827 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:49:38,593 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:49:38,599 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:49:38,601 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:49:38,605 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:49:38,607 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:49:38,611 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:49:38,614 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:49:38,618 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:49:38,620 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:49:38,722 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:49:38,724 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:49:38,727 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:49:38,729 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:49:38,732 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:49:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:43.935629+00:00 (in 5.200061 seconds) notificationworker stdout | 2023-01-17 05:49:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:48 UTC)" (scheduled at 2023-01-17 05:49:38.735066+00:00) notificationworker stdout | 2023-01-17 05:49:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 38, 736028), True, datetime.datetime(2023, 1, 17, 5, 49, 38, 736028), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:48 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:49:38,982 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:49:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:49:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:48.738458+00:00 (in 9.707211 seconds) repositorygcworker stdout | 2023-01-17 05:49:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:39 UTC)" (scheduled at 2023-01-17 05:49:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:49:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:49:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 39, 31723), True, datetime.datetime(2023, 1, 17, 5, 49, 39, 31723), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:49:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:49:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:49:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:52.837365+00:00 (in 13.606872 seconds) securityworker stdout | 2023-01-17 05:49:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:09 UTC)" (scheduled at 2023-01-17 05:49:39.230038+00:00) securityworker stdout | 2023-01-17 05:49:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:49:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:49:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:39,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:49:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:49:39,243 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:49:39,243 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:39,243 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:39,243 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:39,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:49:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:39,600 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:49:40,365 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:49:40,381 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:49:40,389 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:49:40,501 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:49:43,265 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:43,274 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:43,275 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:43,280 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:48.735066+00:00 (in 4.798807 seconds) notificationworker stdout | 2023-01-17 05:49:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:43 UTC)" (scheduled at 2023-01-17 05:49:43.935629+00:00) notificationworker stdout | 2023-01-17 05:49:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:49:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:49:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:49:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:59.337413+00:00 (in 11.404445 seconds) namespacegcworker stdout | 2023-01-17 05:49:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:47 UTC)" (scheduled at 2023-01-17 05:49:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:49:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:49:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:58.735066+00:00 (in 9.999458 seconds) notificationworker stdout | 2023-01-17 05:49:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:58 UTC)" (scheduled at 2023-01-17 05:49:48.735066+00:00) notificationworker stdout | 2023-01-17 05:49:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 48, 736065), True, datetime.datetime(2023, 1, 17, 5, 49, 48, 736065), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:49:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:49:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:39.030814+00:00 (in 50.291905 seconds) repositorygcworker stdout | 2023-01-17 05:49:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:48 UTC)" (scheduled at 2023-01-17 05:49:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:49:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:49:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:49:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:49:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:24.641816+00:00 (in 35.811085 seconds) exportactionlogsworker stdout | 2023-01-17 05:49:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:48 UTC)" (scheduled at 2023-01-17 05:49:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:49:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:49:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 48, 831398), True, datetime.datetime(2023, 1, 17, 5, 49, 48, 831398), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:49:48,839 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:49:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:49:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:49:52,626 [239] [DEBUG] [app] Starting request: urn:request:8ad2f902-4151-47cc-8058-27300a7afe1e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:52,626 [244] [DEBUG] [app] Starting request: urn:request:58091f85-b017-45f9-b0b1-f5dbad6bd13c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:49:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:52,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:52,631 [232] [DEBUG] [app] Starting request: urn:request:97c45b72-7f5a-4078-84c5-3046caf8ffdd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:52,632 [242] [DEBUG] [app] Starting request: urn:request:fda3a41f-4a19-4808-9a52-996608a8b9f1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:52,632 [232] [DEBUG] [app] Ending request: urn:request:97c45b72-7f5a-4078-84c5-3046caf8ffdd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97c45b72-7f5a-4078-84c5-3046caf8ffdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:52,632 [242] [DEBUG] [app] Ending request: urn:request:fda3a41f-4a19-4808-9a52-996608a8b9f1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fda3a41f-4a19-4808-9a52-996608a8b9f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:49:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:49:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:52,636 [243] [DEBUG] [app] Starting request: urn:request:9f5520ee-097a-4c34-8ce5-5734a057190d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:52,636 [239] [DEBUG] [app] Starting request: urn:request:117c8734-f28b-45d2-b501-35fea0a9585f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:52,636 [243] [DEBUG] [app] Ending request: urn:request:9f5520ee-097a-4c34-8ce5-5734a057190d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f5520ee-097a-4c34-8ce5-5734a057190d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:52,637 [239] [DEBUG] [app] Ending request: urn:request:117c8734-f28b-45d2-b501-35fea0a9585f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:117c8734-f28b-45d2-b501-35fea0a9585f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:52,648 [244] [DEBUG] [app] Ending request: urn:request:58091f85-b017-45f9-b0b1-f5dbad6bd13c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:58091f85-b017-45f9-b0b1-f5dbad6bd13c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:52,648 [239] [DEBUG] [app] Ending request: urn:request:8ad2f902-4151-47cc-8058-27300a7afe1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ad2f902-4151-47cc-8058-27300a7afe1e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:49:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:49:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:09.230038+00:00 (in 16.392225 seconds) securityworker stdout | 2023-01-17 05:49:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:22 UTC)" (scheduled at 2023-01-17 05:49:52.837365+00:00) securityworker stdout | 2023-01-17 05:49:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:22 UTC)" executed successfully builder stdout | 2023-01-17 05:49:53,281 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:53,290 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:53,290 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:53,296 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:49:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:49:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:27.226424+00:00 (in 29.999565 seconds) buildlogsarchiver stdout | 2023-01-17 05:49:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:27 UTC)" (scheduled at 2023-01-17 05:49:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:49:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 49, 57, 227228), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:49:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:49:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:49:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:08.735066+00:00 (in 9.999475 seconds) notificationworker stdout | 2023-01-17 05:49:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:08 UTC)" (scheduled at 2023-01-17 05:49:58.735066+00:00) notificationworker stdout | 2023-01-17 05:49:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 58, 736051), True, datetime.datetime(2023, 1, 17, 5, 49, 58, 736051), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:49:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:49:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:47.932490+00:00 (in 48.594649 seconds) namespacegcworker stdout | 2023-01-17 05:49:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:59 UTC)" (scheduled at 2023-01-17 05:49:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:49:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:49:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 59, 338243), True, datetime.datetime(2023, 1, 17, 5, 49, 59, 338243), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:49:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:49:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:49:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:01,455 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:50:01,583 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:50:01,918 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:50:02,644 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:50:02,790 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:50:03,306 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:03,315 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:03,315 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:03,320 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:50:03,993 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:50:04,710 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:50:04,830 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:50:05,096 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:50:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:50:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:35.848675+00:00 (in 29.999498 seconds) gcworker stdout | 2023-01-17 05:50:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:35 UTC)" (scheduled at 2023-01-17 05:50:05.848675+00:00) gcworker stdout | 2023-01-17 05:50:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725005849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:50:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:50:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:50:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:35 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:50:06,124 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:50:06,161 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:50:06,192 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:50:06,198 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:50:06,200 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:50:07,046 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:50:07,625 [244] [DEBUG] [app] Starting request: urn:request:909a6c0b-475e-402e-83b0-246446fa4c82 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:07,625 [239] [DEBUG] [app] Starting request: urn:request:a5c81d97-9e9b-4573-971e-cfa0b596d535 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:07,630 [232] [DEBUG] [app] Starting request: urn:request:f1676057-9a4f-41af-a1bb-9df5926efc18 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,631 [232] [DEBUG] [app] Ending request: urn:request:f1676057-9a4f-41af-a1bb-9df5926efc18 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1676057-9a4f-41af-a1bb-9df5926efc18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,631 [242] [DEBUG] [app] Starting request: urn:request:103b0487-f68c-4464-b33d-21bfba4cad64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:50:07,631 [242] [DEBUG] [app] Ending request: urn:request:103b0487-f68c-4464-b33d-21bfba4cad64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:103b0487-f68c-4464-b33d-21bfba4cad64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:50:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:50:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:07,636 [243] [DEBUG] [app] Starting request: urn:request:beb23c83-6e41-4e6d-b99b-b01d17e9436a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:07,637 [243] [DEBUG] [app] Ending request: urn:request:beb23c83-6e41-4e6d-b99b-b01d17e9436a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:beb23c83-6e41-4e6d-b99b-b01d17e9436a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:07,637 [241] [DEBUG] [app] Starting request: urn:request:5cad06aa-5c9c-4879-a939-aba5e2524259 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:07,637 [241] [DEBUG] [app] Ending request: urn:request:5cad06aa-5c9c-4879-a939-aba5e2524259 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5cad06aa-5c9c-4879-a939-aba5e2524259', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:07,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:07,648 [239] [DEBUG] [app] Ending request: urn:request:a5c81d97-9e9b-4573-971e-cfa0b596d535 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5c81d97-9e9b-4573-971e-cfa0b596d535', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:07,648 [244] [DEBUG] [app] Ending request: urn:request:909a6c0b-475e-402e-83b0-246446fa4c82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:909a6c0b-475e-402e-83b0-246446fa4c82', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:07,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:50:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:50:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:50:07,829 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:50:07,862 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:50:08,604 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:50:08,606 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:50:08,610 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:50:08,613 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:50:08,615 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:50:08,619 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:50:08,625 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:50:08,627 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:50:08,629 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:50:08,728 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:50:08,731 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:50:08,733 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} notificationworker stdout | 2023-01-17 05:50:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:18.735066+00:00 (in 9.999499 seconds) gunicorn-web stdout | 2023-01-17 05:50:08,735 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:50:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:18 UTC)" (scheduled at 2023-01-17 05:50:08.735066+00:00) notificationworker stdout | 2023-01-17 05:50:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:08,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 8, 736031), True, datetime.datetime(2023, 1, 17, 5, 50, 8, 736031), 0, 'notification/%', 50, 1, 0]) gunicorn-web stdout | 2023-01-17 05:50:08,739 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:50:08,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:08,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:08,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:18 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:50:08,992 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:50:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:22.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 05:50:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:39 UTC)" (scheduled at 2023-01-17 05:50:09.230038+00:00) securityworker stdout | 2023-01-17 05:50:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:50:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:50:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:09,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:50:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:50:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:50:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:09,243 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:09,243 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:09,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:50:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:09,635 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:50:10,385 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:50:10,393 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:50:10,396 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:50:10,537 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:50:13,331 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:13,340 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:13,340 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:13,346 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:28.735066+00:00 (in 9.999374 seconds) notificationworker stdout | 2023-01-17 05:50:18,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:28 UTC)" (scheduled at 2023-01-17 05:50:18.735066+00:00) notificationworker stdout | 2023-01-17 05:50:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 18, 736140), True, datetime.datetime(2023, 1, 17, 5, 50, 18, 736140), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:50:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:50:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:25.334138+00:00 (in 5.091524 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:50:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:20 UTC)" (scheduled at 2023-01-17 05:50:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:50:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:50:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 20, 243049), True, datetime.datetime(2023, 1, 17, 5, 50, 20, 243049), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:50:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:50:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:50:20,252 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:50:22,625 [239] [DEBUG] [app] Starting request: urn:request:37501db4-a2aa-4bdf-9ad5-47812e534585 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:22,625 [243] [DEBUG] [app] Starting request: urn:request:3984ccc8-5cf7-45aa-ba55-45b044a7d4c2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:22,630 [232] [DEBUG] [app] Starting request: urn:request:b1116ddc-4e2d-45b6-81df-46707bde08cf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:22,630 [232] [DEBUG] [app] Ending request: urn:request:b1116ddc-4e2d-45b6-81df-46707bde08cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1116ddc-4e2d-45b6-81df-46707bde08cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:22,630 [242] [DEBUG] [app] Starting request: urn:request:d53edf3e-66b5-49aa-9edb-a22c64e99747 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:50:22,631 [242] [DEBUG] [app] Ending request: urn:request:d53edf3e-66b5-49aa-9edb-a22c64e99747 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d53edf3e-66b5-49aa-9edb-a22c64e99747', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:50:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:50:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:22,634 [239] [DEBUG] [app] Starting request: urn:request:c55b2de0-b232-47c9-bdc9-d946ced59031 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:22,635 [239] [DEBUG] [app] Ending request: urn:request:c55b2de0-b232-47c9-bdc9-d946ced59031 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c55b2de0-b232-47c9-bdc9-d946ced59031', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:22,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:22,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:22,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:22,636 [244] [DEBUG] [app] Starting request: urn:request:76f88ee3-fd14-4b3a-a8b9-e4f93215498d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:22,637 [244] [DEBUG] [app] Ending request: urn:request:76f88ee3-fd14-4b3a-a8b9-e4f93215498d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76f88ee3-fd14-4b3a-a8b9-e4f93215498d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:22,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:22,646 [243] [DEBUG] [app] Ending request: urn:request:3984ccc8-5cf7-45aa-ba55-45b044a7d4c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3984ccc8-5cf7-45aa-ba55-45b044a7d4c2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:22,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:22,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:50:22,647 [239] [DEBUG] [app] Ending request: urn:request:37501db4-a2aa-4bdf-9ad5-47812e534585 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37501db4-a2aa-4bdf-9ad5-47812e534585', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:50:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:39.230038+00:00 (in 16.392201 seconds) securityworker stdout | 2023-01-17 05:50:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:52 UTC)" (scheduled at 2023-01-17 05:50:22.837365+00:00) securityworker stdout | 2023-01-17 05:50:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:52 UTC)" executed successfully builder stdout | 2023-01-17 05:50:23,354 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:23,362 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:23,362 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:23,368 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:50:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:50:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:48.830274+00:00 (in 24.188037 seconds) exportactionlogsworker stdout | 2023-01-17 05:50:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:24 UTC)" (scheduled at 2023-01-17 05:50:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:50:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:50:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:50:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:50:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:20.242198+00:00 (in 54.907596 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:50:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:25 UTC)" (scheduled at 2023-01-17 05:50:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:50:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:50:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:50:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:50:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:57.226424+00:00 (in 29.999566 seconds) buildlogsarchiver stdout | 2023-01-17 05:50:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:57 UTC)" (scheduled at 2023-01-17 05:50:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:50:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 50, 27, 227227), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:50:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:50:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:50:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:38.735066+00:00 (in 9.999431 seconds) notificationworker stdout | 2023-01-17 05:50:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:38 UTC)" (scheduled at 2023-01-17 05:50:28.735066+00:00) notificationworker stdout | 2023-01-17 05:50:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 28, 736111), True, datetime.datetime(2023, 1, 17, 5, 50, 28, 736111), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:31,483 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:50:31,617 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:50:31,934 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:50:32,676 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:50:32,825 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:50:33,378 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:33,387 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:33,387 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:33,392 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:50:34,020 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:50:34,746 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:50:34,846 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:50:35,132 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:50:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:50:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:05.848675+00:00 (in 29.999536 seconds) gcworker stdout | 2023-01-17 05:50:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:05 UTC)" (scheduled at 2023-01-17 05:50:35.848675+00:00) gcworker stdout | 2023-01-17 05:50:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725035849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:50:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:50:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:50:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:05 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:50:36,142 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:50:36,180 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-secscan stdout | 2023-01-17 05:50:36,198 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:50:36,203 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:50:36,206 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} chunkcleanupworker stdout | 2023-01-17 05:50:37,079 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:50:37,626 [244] [DEBUG] [app] Starting request: urn:request:1beeb3e6-f415-4986-abad-1489067daa6a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:37,626 [241] [DEBUG] [app] Starting request: urn:request:cc432940-029a-4f34-9efc-50842b9f63a9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:37,630 [240] [DEBUG] [app] Starting request: urn:request:0a3122f6-9da1-4611-8b92-683292899355 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:37,630 [240] [DEBUG] [app] Ending request: urn:request:0a3122f6-9da1-4611-8b92-683292899355 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a3122f6-9da1-4611-8b92-683292899355', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:37,631 [242] [DEBUG] [app] Starting request: urn:request:b9676818-5e63-4281-af97-5a179dc2eb69 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:50:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:50:37,631 [242] [DEBUG] [app] Ending request: urn:request:b9676818-5e63-4281-af97-5a179dc2eb69 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9676818-5e63-4281-af97-5a179dc2eb69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:50:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:37,636 [239] [DEBUG] [app] Starting request: urn:request:6224de32-4384-4724-81f6-4bdc938fee76 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:37,636 [244] [DEBUG] [app] Starting request: urn:request:5ad1a576-0bf2-4967-93a6-eccfdf64ca22 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:37,636 [239] [DEBUG] [app] Ending request: urn:request:6224de32-4384-4724-81f6-4bdc938fee76 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6224de32-4384-4724-81f6-4bdc938fee76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:37,636 [244] [DEBUG] [app] Ending request: urn:request:5ad1a576-0bf2-4967-93a6-eccfdf64ca22 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ad1a576-0bf2-4967-93a6-eccfdf64ca22', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:37,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:37,648 [244] [DEBUG] [app] Ending request: urn:request:1beeb3e6-f415-4986-abad-1489067daa6a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1beeb3e6-f415-4986-abad-1489067daa6a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:37,648 [241] [DEBUG] [app] Ending request: urn:request:cc432940-029a-4f34-9efc-50842b9f63a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc432940-029a-4f34-9efc-50842b9f63a9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:50:37,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) logrotateworker stdout | 2023-01-17 05:50:37,845 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:50:37,866 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:50:38,614 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:50:38,616 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:50:38,618 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:50:38,623 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:50:38,626 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:50:38,628 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:50:38,633 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:50:38,635 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:50:38,637 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:50:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gunicorn-web stdout | 2023-01-17 05:50:38,735 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} notificationworker stdout | 2023-01-17 05:50:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:43.935629+00:00 (in 5.200029 seconds) notificationworker stdout | 2023-01-17 05:50:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:48 UTC)" (scheduled at 2023-01-17 05:50:38.735066+00:00) notificationworker stdout | 2023-01-17 05:50:38,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 38, 736030), True, datetime.datetime(2023, 1, 17, 5, 50, 38, 736030), 0, 'notification/%', 50, 1, 0]) gunicorn-web stdout | 2023-01-17 05:50:38,738 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:50:38,741 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:50:38,743 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:50:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:50:38,746 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:50:39,028 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:50:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:50:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:48.738458+00:00 (in 9.707236 seconds) repositorygcworker stdout | 2023-01-17 05:50:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:39 UTC)" (scheduled at 2023-01-17 05:50:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:50:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:50:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 39, 31627), True, datetime.datetime(2023, 1, 17, 5, 50, 39, 31627), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:50:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:50:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:50:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:52.837365+00:00 (in 13.606876 seconds) securityworker stdout | 2023-01-17 05:50:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:09 UTC)" (scheduled at 2023-01-17 05:50:39.230038+00:00) securityworker stdout | 2023-01-17 05:50:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:50:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:50:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:39,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:50:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:50:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:50:39,243 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:39,243 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:39,243 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:39,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:50:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:39,671 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:50:40,405 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:50:40,426 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:50:40,430 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:50:40,563 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:50:43,403 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:43,412 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:43,412 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:43,418 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:48.735066+00:00 (in 4.798917 seconds) notificationworker stdout | 2023-01-17 05:50:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:43 UTC)" (scheduled at 2023-01-17 05:50:43.935629+00:00) notificationworker stdout | 2023-01-17 05:50:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:50:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:50:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:50:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:59.337413+00:00 (in 11.404452 seconds) namespacegcworker stdout | 2023-01-17 05:50:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:47 UTC)" (scheduled at 2023-01-17 05:50:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:50:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:50:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:58.735066+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:50:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:58 UTC)" (scheduled at 2023-01-17 05:50:48.735066+00:00) notificationworker stdout | 2023-01-17 05:50:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 48, 735928), True, datetime.datetime(2023, 1, 17, 5, 50, 48, 735928), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:50:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:50:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:39.030814+00:00 (in 50.291905 seconds) repositorygcworker stdout | 2023-01-17 05:50:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:48 UTC)" (scheduled at 2023-01-17 05:50:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:50:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:50:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:50:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:50:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:24.641816+00:00 (in 35.811074 seconds) exportactionlogsworker stdout | 2023-01-17 05:50:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:48 UTC)" (scheduled at 2023-01-17 05:50:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:50:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:50:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 48, 831091), True, datetime.datetime(2023, 1, 17, 5, 50, 48, 831091), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:50:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:50:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:50:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:50:52,625 [239] [DEBUG] [app] Starting request: urn:request:295624e9-f904-4697-86f2-7709dcadd7cd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,625 [243] [DEBUG] [app] Starting request: urn:request:29b79c8c-146d-4f64-9576-f9119cd92d4c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:52,630 [242] [DEBUG] [app] Starting request: urn:request:d7dc2211-273f-475d-b2ad-e41c260ebe81 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,630 [242] [DEBUG] [app] Ending request: urn:request:d7dc2211-273f-475d-b2ad-e41c260ebe81 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d7dc2211-273f-475d-b2ad-e41c260ebe81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,631 [240] [DEBUG] [app] Starting request: urn:request:13d513ed-dfad-479a-9e73-9aad42a85a07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:50:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:50:52,631 [240] [DEBUG] [app] Ending request: urn:request:13d513ed-dfad-479a-9e73-9aad42a85a07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13d513ed-dfad-479a-9e73-9aad42a85a07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.003) gunicorn-web stdout | 2023-01-17 05:50:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:52,635 [244] [DEBUG] [app] Starting request: urn:request:4b6aae5f-6a59-43e4-9a79-e67b8032df17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,635 [244] [DEBUG] [app] Ending request: urn:request:4b6aae5f-6a59-43e4-9a79-e67b8032df17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b6aae5f-6a59-43e4-9a79-e67b8032df17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:52,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:52,636 [244] [DEBUG] [app] Starting request: urn:request:ffc27d12-ab20-45b8-a7bc-cc9a77d592df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,637 [244] [DEBUG] [app] Ending request: urn:request:ffc27d12-ab20-45b8-a7bc-cc9a77d592df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ffc27d12-ab20-45b8-a7bc-cc9a77d592df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:52,646 [239] [DEBUG] [app] Ending request: urn:request:295624e9-f904-4697-86f2-7709dcadd7cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:295624e9-f904-4697-86f2-7709dcadd7cd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:52,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:50:52,647 [243] [DEBUG] [app] Ending request: urn:request:29b79c8c-146d-4f64-9576-f9119cd92d4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29b79c8c-146d-4f64-9576-f9119cd92d4c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:50:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:09.230038+00:00 (in 16.392253 seconds) securityworker stdout | 2023-01-17 05:50:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:22 UTC)" (scheduled at 2023-01-17 05:50:52.837365+00:00) securityworker stdout | 2023-01-17 05:50:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:52,844 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:22 UTC)" executed successfully builder stdout | 2023-01-17 05:50:53,426 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:53,434 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:53,434 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:53,440 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:50:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:50:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:27.226424+00:00 (in 29.999577 seconds) buildlogsarchiver stdout | 2023-01-17 05:50:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:27 UTC)" (scheduled at 2023-01-17 05:50:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:50:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 50, 57, 227221), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:50:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:50:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:50:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:08.735066+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 05:50:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:08 UTC)" (scheduled at 2023-01-17 05:50:58.735066+00:00) notificationworker stdout | 2023-01-17 05:50:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 58, 735924), True, datetime.datetime(2023, 1, 17, 5, 50, 58, 735924), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:50:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:50:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:47.932490+00:00 (in 48.594617 seconds) namespacegcworker stdout | 2023-01-17 05:50:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:59 UTC)" (scheduled at 2023-01-17 05:50:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:50:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:50:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 59, 338254), True, datetime.datetime(2023, 1, 17, 5, 50, 59, 338254), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:50:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:50:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:50:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:01,510 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:51:01,652 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:51:01,966 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:51:02,705 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:51:02,837 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:51:03,441 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:03,450 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:03,450 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:03,455 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:51:04,036 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:51:04,781 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:51:04,881 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:51:05,166 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:51:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:51:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:35.848675+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 05:51:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:35 UTC)" (scheduled at 2023-01-17 05:51:05.848675+00:00) gcworker stdout | 2023-01-17 05:51:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725065849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:51:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:51:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:51:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:35 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:51:06,167 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:51:06,204 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:51:06,209 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:51:06,214 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:51:06,216 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:51:07,113 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:51:07,625 [243] [DEBUG] [app] Starting request: urn:request:406d991d-14f8-4df4-9ea1-ec63a3440d6a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:07,625 [239] [DEBUG] [app] Starting request: urn:request:5248d73b-06c6-4688-bd98-6461094ccd56 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:07,630 [232] [DEBUG] [app] Starting request: urn:request:e4cd450a-efaf-4220-aa5f-c3eec42acda2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,631 [242] [DEBUG] [app] Starting request: urn:request:e5651c38-8615-46be-b2cb-4df8c1f6db5b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,631 [232] [DEBUG] [app] Ending request: urn:request:e4cd450a-efaf-4220-aa5f-c3eec42acda2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e4cd450a-efaf-4220-aa5f-c3eec42acda2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:51:07,631 [242] [DEBUG] [app] Ending request: urn:request:e5651c38-8615-46be-b2cb-4df8c1f6db5b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5651c38-8615-46be-b2cb-4df8c1f6db5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:51:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:51:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:07,635 [243] [DEBUG] [app] Starting request: urn:request:6e3c741c-a435-4253-a799-239e19ac8b5d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:07,636 [243] [DEBUG] [app] Ending request: urn:request:6e3c741c-a435-4253-a799-239e19ac8b5d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e3c741c-a435-4253-a799-239e19ac8b5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:51:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:07,637 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:07,638 [244] [DEBUG] [app] Starting request: urn:request:f7504c91-2e33-403c-9a10-c6bbfbfad46a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:07,638 [244] [DEBUG] [app] Ending request: urn:request:f7504c91-2e33-403c-9a10-c6bbfbfad46a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7504c91-2e33-403c-9a10-c6bbfbfad46a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:07,639 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:07,639 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:07,639 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:07,644 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:07,647 [239] [DEBUG] [app] Ending request: urn:request:5248d73b-06c6-4688-bd98-6461094ccd56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5248d73b-06c6-4688-bd98-6461094ccd56', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:51:07,649 [243] [DEBUG] [app] Ending request: urn:request:406d991d-14f8-4df4-9ea1-ec63a3440d6a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:406d991d-14f8-4df4-9ea1-ec63a3440d6a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:07,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) logrotateworker stdout | 2023-01-17 05:51:07,868 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:51:07,880 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:51:08,621 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:51:08,625 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:51:08,628 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:51:08,634 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:51:08,636 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:51:08,638 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:51:08,641 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:51:08,644 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:51:08,646 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:51:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:18.735066+00:00 (in 9.999562 seconds) notificationworker stdout | 2023-01-17 05:51:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:18 UTC)" (scheduled at 2023-01-17 05:51:08.735066+00:00) notificationworker stdout | 2023-01-17 05:51:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 8, 735877), True, datetime.datetime(2023, 1, 17, 5, 51, 8, 735877), 0, 'notification/%', 50, 1, 0]) gunicorn-web stdout | 2023-01-17 05:51:08,741 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:51:08,744 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} notificationworker stdout | 2023-01-17 05:51:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:51:08,746 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:51:08,749 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:51:08,752 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:51:09,059 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:51:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:22.837365+00:00 (in 13.606904 seconds) securityworker stdout | 2023-01-17 05:51:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:39 UTC)" (scheduled at 2023-01-17 05:51:09.230038+00:00) securityworker stdout | 2023-01-17 05:51:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:51:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:51:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:09,245 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:09,245 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:51:09,246 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:09,689 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:51:10,423 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:51:10,461 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:51:10,464 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:51:10,584 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:51:13,466 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:13,475 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:13,475 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:13,481 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:28.735066+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 05:51:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:28 UTC)" (scheduled at 2023-01-17 05:51:18.735066+00:00) notificationworker stdout | 2023-01-17 05:51:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 18, 735877), True, datetime.datetime(2023, 1, 17, 5, 51, 18, 735877), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:51:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:51:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:25.334138+00:00 (in 5.091490 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:51:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:20 UTC)" (scheduled at 2023-01-17 05:51:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:51:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:51:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 20, 243061), True, datetime.datetime(2023, 1, 17, 5, 51, 20, 243061), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:51:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:51:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:51:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:51:22,625 [244] [DEBUG] [app] Starting request: urn:request:05589c69-871e-4cb4-b0f2-63c1c8ae6de7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:22,626 [239] [DEBUG] [app] Starting request: urn:request:9d33ef7f-1a08-4de9-8e7f-6a998aff6552 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:22,630 [232] [DEBUG] [app] Starting request: urn:request:82e09a2a-c958-48e3-a7fa-bf08588547c5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:22,630 [240] [DEBUG] [app] Starting request: urn:request:61c4d773-497b-42e1-aa06-2e5448a8a951 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:22,631 [232] [DEBUG] [app] Ending request: urn:request:82e09a2a-c958-48e3-a7fa-bf08588547c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82e09a2a-c958-48e3-a7fa-bf08588547c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:51:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:51:22,631 [240] [DEBUG] [app] Ending request: urn:request:61c4d773-497b-42e1-aa06-2e5448a8a951 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:61c4d773-497b-42e1-aa06-2e5448a8a951', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:51:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:51:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:22,635 [241] [DEBUG] [app] Starting request: urn:request:451efa49-0f99-44ce-a54e-b58d18e2ec4c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:22,635 [243] [DEBUG] [app] Starting request: urn:request:bf5ecf4c-e8d5-47aa-a6d1-6bceab42fc6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:22,635 [241] [DEBUG] [app] Ending request: urn:request:451efa49-0f99-44ce-a54e-b58d18e2ec4c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:451efa49-0f99-44ce-a54e-b58d18e2ec4c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:22,636 [243] [DEBUG] [app] Ending request: urn:request:bf5ecf4c-e8d5-47aa-a6d1-6bceab42fc6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf5ecf4c-e8d5-47aa-a6d1-6bceab42fc6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:51:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:22,647 [239] [DEBUG] [app] Ending request: urn:request:9d33ef7f-1a08-4de9-8e7f-6a998aff6552 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d33ef7f-1a08-4de9-8e7f-6a998aff6552', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:51:22,650 [244] [DEBUG] [app] Ending request: urn:request:05589c69-871e-4cb4-b0f2-63c1c8ae6de7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05589c69-871e-4cb4-b0f2-63c1c8ae6de7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:22,650 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:22,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) securityworker stdout | 2023-01-17 05:51:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:39.230038+00:00 (in 16.392185 seconds) securityworker stdout | 2023-01-17 05:51:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:52 UTC)" (scheduled at 2023-01-17 05:51:22.837365+00:00) securityworker stdout | 2023-01-17 05:51:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:52 UTC)" executed successfully builder stdout | 2023-01-17 05:51:23,491 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:23,500 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:23,500 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:23,506 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:51:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:51:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:44.641497+00:00 (in 19.999232 seconds) exportactionlogsworker stdout | 2023-01-17 05:51:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:24 UTC)" (scheduled at 2023-01-17 05:51:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:51:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:51:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:51:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:51:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:39.333860+00:00 (in 13.999247 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:51:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:25 UTC)" (scheduled at 2023-01-17 05:51:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:51:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:51:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:51:27,227 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:51:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:57.226424+00:00 (in 29.999274 seconds) buildlogsarchiver stdout | 2023-01-17 05:51:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:57 UTC)" (scheduled at 2023-01-17 05:51:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:51:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 51, 27, 227521), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:51:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:51:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:51:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:38.735066+00:00 (in 9.999582 seconds) notificationworker stdout | 2023-01-17 05:51:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:38 UTC)" (scheduled at 2023-01-17 05:51:28.735066+00:00) notificationworker stdout | 2023-01-17 05:51:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 28, 735854), True, datetime.datetime(2023, 1, 17, 5, 51, 28, 735854), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:31,536 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:51:31,688 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:51:32,002 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:51:32,741 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:51:32,872 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:51:33,516 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:33,524 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:33,524 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:33,530 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:51:34,055 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:51:34,787 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:51:34,887 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:51:35,182 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:51:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:51:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:05.848675+00:00 (in 29.999538 seconds) gcworker stdout | 2023-01-17 05:51:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:05 UTC)" (scheduled at 2023-01-17 05:51:35.848675+00:00) gcworker stdout | 2023-01-17 05:51:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725095849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:51:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:51:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:51:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:05 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:51:36,203 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:51:36,210 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:51:36,216 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:51:36,219 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:51:36,250 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:51:37,126 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:51:37,625 [244] [DEBUG] [app] Starting request: urn:request:65e7f3dd-b5c9-41c9-ad26-d4e90cb1f529 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:37,625 [241] [DEBUG] [app] Starting request: urn:request:2cbc7412-8856-4e74-80d8-f03686930c3c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:37,630 [242] [DEBUG] [app] Starting request: urn:request:a049c6a3-c0da-47c5-bb65-48756dc45d35 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:37,630 [242] [DEBUG] [app] Ending request: urn:request:a049c6a3-c0da-47c5-bb65-48756dc45d35 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a049c6a3-c0da-47c5-bb65-48756dc45d35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:37,630 [240] [DEBUG] [app] Starting request: urn:request:30428dd9-8319-4f8b-9319-7377a50241db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:37,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:51:37,631 [240] [DEBUG] [app] Ending request: urn:request:30428dd9-8319-4f8b-9319-7377a50241db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30428dd9-8319-4f8b-9319-7377a50241db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:51:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:51:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:37,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:37,635 [243] [DEBUG] [app] Starting request: urn:request:2254a021-67cf-4650-bd93-e6e733a9cf74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:37,635 [239] [DEBUG] [app] Starting request: urn:request:97307e15-f3e0-4ffe-9628-ea0a439ad9e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:37,635 [243] [DEBUG] [app] Ending request: urn:request:2254a021-67cf-4650-bd93-e6e733a9cf74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2254a021-67cf-4650-bd93-e6e733a9cf74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:37,635 [239] [DEBUG] [app] Ending request: urn:request:97307e15-f3e0-4ffe-9628-ea0a439ad9e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97307e15-f3e0-4ffe-9628-ea0a439ad9e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:37,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:37,647 [241] [DEBUG] [app] Ending request: urn:request:2cbc7412-8856-4e74-80d8-f03686930c3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cbc7412-8856-4e74-80d8-f03686930c3c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:37,647 [244] [DEBUG] [app] Ending request: urn:request:65e7f3dd-b5c9-41c9-ad26-d4e90cb1f529 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65e7f3dd-b5c9-41c9-ad26-d4e90cb1f529', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:51:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:51:37,903 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:51:37,915 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:51:38,629 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:51:38,635 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:51:38,638 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:51:38,642 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:51:38,644 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:51:38,646 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:51:38,649 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:51:38,652 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:51:38,654 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:51:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:43.935629+00:00 (in 5.200164 seconds) notificationworker stdout | 2023-01-17 05:51:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:48 UTC)" (scheduled at 2023-01-17 05:51:38.735066+00:00) notificationworker stdout | 2023-01-17 05:51:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 38, 735818), True, datetime.datetime(2023, 1, 17, 5, 51, 38, 735818), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:51:38,747 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:51:38,749 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:51:38,753 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:51:38,756 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:51:38,759 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:51:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:51:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:48.738458+00:00 (in 9.707231 seconds) repositorygcworker stdout | 2023-01-17 05:51:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:39 UTC)" (scheduled at 2023-01-17 05:51:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:51:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:51:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 39, 31654), True, datetime.datetime(2023, 1, 17, 5, 51, 39, 31654), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:51:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:51:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:51:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:51:39,096 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:51:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:52.837365+00:00 (in 13.606867 seconds) securityworker stdout | 2023-01-17 05:51:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:09 UTC)" (scheduled at 2023-01-17 05:51:39.230038+00:00) securityworker stdout | 2023-01-17 05:51:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:39,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:51:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:51:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:51:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:51:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:51:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:20.242198+00:00 (in 40.907906 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:51:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:39 UTC)" (scheduled at 2023-01-17 05:51:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:51:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 51, 39, 334658), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:51:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 39, 334658), True, datetime.datetime(2023, 1, 17, 5, 51, 39, 334658), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:51:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 39, 334658), True, datetime.datetime(2023, 1, 17, 5, 51, 39, 334658), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 51, 39, 334658), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:51:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:51:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:39,697 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:51:40,434 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:51:40,476 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:51:40,498 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:51:40,619 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:51:43,540 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:43,549 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:43,549 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:43,555 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:48.735066+00:00 (in 4.798997 seconds) notificationworker stdout | 2023-01-17 05:51:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:43 UTC)" (scheduled at 2023-01-17 05:51:43.935629+00:00) notificationworker stdout | 2023-01-17 05:51:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:51:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:51:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:51:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:48.830274+00:00 (in 4.188341 seconds) exportactionlogsworker stdout | 2023-01-17 05:51:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:44 UTC)" (scheduled at 2023-01-17 05:51:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:51:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 51, 44, 642336), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:51:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 44, 642336), True, datetime.datetime(2023, 1, 17, 5, 51, 44, 642336), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:51:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 44, 642336), True, datetime.datetime(2023, 1, 17, 5, 51, 44, 642336), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 51, 44, 642336), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:51:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:51:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:51:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:51:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:59.337413+00:00 (in 11.404446 seconds) namespacegcworker stdout | 2023-01-17 05:51:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:47 UTC)" (scheduled at 2023-01-17 05:51:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:51:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:51:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:58.735066+00:00 (in 9.999573 seconds) notificationworker stdout | 2023-01-17 05:51:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:58 UTC)" (scheduled at 2023-01-17 05:51:48.735066+00:00) notificationworker stdout | 2023-01-17 05:51:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 48, 735881), True, datetime.datetime(2023, 1, 17, 5, 51, 48, 735881), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:51:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:51:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:04.738062+00:00 (in 15.999127 seconds) repositorygcworker stdout | 2023-01-17 05:51:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:48 UTC)" (scheduled at 2023-01-17 05:51:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:51:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:51:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:51:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:51:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:24.641816+00:00 (in 35.811123 seconds) exportactionlogsworker stdout | 2023-01-17 05:51:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:48 UTC)" (scheduled at 2023-01-17 05:51:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:51:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:51:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 48, 831055), True, datetime.datetime(2023, 1, 17, 5, 51, 48, 831055), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:51:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:51:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:51:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:51:52,625 [243] [DEBUG] [app] Starting request: urn:request:0c12a018-6656-44b9-9204-010ec6b22ac4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:52,625 [241] [DEBUG] [app] Starting request: urn:request:e9c683d9-6f49-40c6-aa1d-6384012bbcd1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:51:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:52,629 [242] [DEBUG] [app] Starting request: urn:request:5c03749d-bdb9-4126-ba1c-e4e4d3000591 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,630 [240] [DEBUG] [app] Starting request: urn:request:5324dc79-6cdf-4ec7-8077-0c593af3d5be (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,630 [242] [DEBUG] [app] Ending request: urn:request:5c03749d-bdb9-4126-ba1c-e4e4d3000591 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c03749d-bdb9-4126-ba1c-e4e4d3000591', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,630 [240] [DEBUG] [app] Ending request: urn:request:5324dc79-6cdf-4ec7-8077-0c593af3d5be (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5324dc79-6cdf-4ec7-8077-0c593af3d5be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:51:52,630 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:51:52,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:52,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:52,631 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,633 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:52,634 [239] [DEBUG] [app] Starting request: urn:request:840cf131-dc8b-4230-815a-ebb261261aa4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:52,635 [239] [DEBUG] [app] Ending request: urn:request:840cf131-dc8b-4230-815a-ebb261261aa4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:840cf131-dc8b-4230-815a-ebb261261aa4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:52,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:52,635 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:52,635 [244] [DEBUG] [app] Starting request: urn:request:18579c37-e664-4e73-b600-3c19bb5161a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:52,635 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:52,636 [244] [DEBUG] [app] Ending request: urn:request:18579c37-e664-4e73-b600-3c19bb5161a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18579c37-e664-4e73-b600-3c19bb5161a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:51:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:52,640 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:52,646 [241] [DEBUG] [app] Ending request: urn:request:e9c683d9-6f49-40c6-aa1d-6384012bbcd1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e9c683d9-6f49-40c6-aa1d-6384012bbcd1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:52,646 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:52,646 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:51:52,646 [243] [DEBUG] [app] Ending request: urn:request:0c12a018-6656-44b9-9204-010ec6b22ac4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c12a018-6656-44b9-9204-010ec6b22ac4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:51:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:51:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:09.230038+00:00 (in 16.392203 seconds) securityworker stdout | 2023-01-17 05:51:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:22 UTC)" (scheduled at 2023-01-17 05:51:52.837365+00:00) securityworker stdout | 2023-01-17 05:51:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:22 UTC)" executed successfully builder stdout | 2023-01-17 05:51:53,565 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:53,575 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:53,575 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:53,581 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:51:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:51:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:27.226424+00:00 (in 29.999559 seconds) buildlogsarchiver stdout | 2023-01-17 05:51:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:27 UTC)" (scheduled at 2023-01-17 05:51:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:51:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 51, 57, 227230), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:51:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:51:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:51:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:08.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 05:51:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:08 UTC)" (scheduled at 2023-01-17 05:51:58.735066+00:00) notificationworker stdout | 2023-01-17 05:51:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 58, 735883), True, datetime.datetime(2023, 1, 17, 5, 51, 58, 735883), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:51:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:51:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:47.932490+00:00 (in 48.594613 seconds) namespacegcworker stdout | 2023-01-17 05:51:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:59 UTC)" (scheduled at 2023-01-17 05:51:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:51:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:51:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 59, 338281), True, datetime.datetime(2023, 1, 17, 5, 51, 59, 338281), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:51:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:51:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:51:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:01,563 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:52:01,723 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:52:02,037 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:52:02,773 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:52:02,878 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:52:03,591 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:03,600 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:03,600 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:03,606 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:52:04,080 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} repositorygcworker stdout | 2023-01-17 05:52:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:52:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:39.030814+00:00 (in 34.292253 seconds) repositorygcworker stdout | 2023-01-17 05:52:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:57:04 UTC)" (scheduled at 2023-01-17 05:52:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:52:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 52, 4, 738937), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:52:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 52, 4, 738937), True, datetime.datetime(2023, 1, 17, 5, 52, 4, 738937), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:52:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 52, 4, 738937), True, datetime.datetime(2023, 1, 17, 5, 52, 4, 738937), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 52, 4, 738937), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:52:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:52:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:57:04 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:52:04,822 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:52:04,923 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:52:05,189 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:52:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:52:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:35.848675+00:00 (in 29.999543 seconds) gcworker stdout | 2023-01-17 05:52:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:35 UTC)" (scheduled at 2023-01-17 05:52:05.848675+00:00) gcworker stdout | 2023-01-17 05:52:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725125849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:52:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:52:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:52:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:52:06,216 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} repositoryactioncounter stdout | 2023-01-17 05:52:06,218 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-secscan stdout | 2023-01-17 05:52:06,221 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:52:06,224 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:52:06,264 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:52:07,159 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:52:07,625 [239] [DEBUG] [app] Starting request: urn:request:4d74a05b-e5d6-45ce-9fd1-12d563a447e8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:07,625 [241] [DEBUG] [app] Starting request: urn:request:2d281ecc-7de9-42e2-bf8f-674dfb05cd2a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:07,630 [242] [DEBUG] [app] Starting request: urn:request:ab538af3-32e5-4aaf-8d64-5c368f33522c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,631 [232] [DEBUG] [app] Starting request: urn:request:b6d55b5c-de0a-4d54-b6d8-d6fab42a8fa1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,631 [242] [DEBUG] [app] Ending request: urn:request:ab538af3-32e5-4aaf-8d64-5c368f33522c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab538af3-32e5-4aaf-8d64-5c368f33522c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:07,631 [232] [DEBUG] [app] Ending request: urn:request:b6d55b5c-de0a-4d54-b6d8-d6fab42a8fa1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6d55b5c-de0a-4d54-b6d8-d6fab42a8fa1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:52:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:52:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:07,636 [244] [DEBUG] [app] Starting request: urn:request:595630d5-915b-43ac-8415-909644252f96 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:07,636 [239] [DEBUG] [app] Starting request: urn:request:e6345bf3-a7df-4935-a327-b3d37b19a74b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:07,636 [244] [DEBUG] [app] Ending request: urn:request:595630d5-915b-43ac-8415-909644252f96 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:595630d5-915b-43ac-8415-909644252f96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:07,636 [239] [DEBUG] [app] Ending request: urn:request:e6345bf3-a7df-4935-a327-b3d37b19a74b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6345bf3-a7df-4935-a327-b3d37b19a74b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:07,647 [239] [DEBUG] [app] Ending request: urn:request:4d74a05b-e5d6-45ce-9fd1-12d563a447e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d74a05b-e5d6-45ce-9fd1-12d563a447e8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:07,647 [241] [DEBUG] [app] Ending request: urn:request:2d281ecc-7de9-42e2-bf8f-674dfb05cd2a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d281ecc-7de9-42e2-bf8f-674dfb05cd2a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:07,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:52:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) logrotateworker stdout | 2023-01-17 05:52:07,934 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:52:07,937 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:52:08,637 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:52:08,644 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:52:08,646 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:52:08,649 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:52:08,651 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:52:08,655 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:52:08,658 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:52:08,660 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:52:08,662 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:52:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:18.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 05:52:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:18 UTC)" (scheduled at 2023-01-17 05:52:08.735066+00:00) notificationworker stdout | 2023-01-17 05:52:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 8, 735921), True, datetime.datetime(2023, 1, 17, 5, 52, 8, 735921), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:08,754 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:52:08,756 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:52:08,759 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:52:08,763 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:52:08,765 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:52:09,131 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:52:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:22.837365+00:00 (in 13.606947 seconds) securityworker stdout | 2023-01-17 05:52:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:39 UTC)" (scheduled at 2023-01-17 05:52:09.230038+00:00) securityworker stdout | 2023-01-17 05:52:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:52:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:52:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:52:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:09,709 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:52:10,470 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:52:10,489 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:52:10,533 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:52:10,652 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:52:13,607 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:13,615 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:13,616 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:13,621 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:28.735066+00:00 (in 9.999537 seconds) notificationworker stdout | 2023-01-17 05:52:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:28 UTC)" (scheduled at 2023-01-17 05:52:18.735066+00:00) notificationworker stdout | 2023-01-17 05:52:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 18, 735944), True, datetime.datetime(2023, 1, 17, 5, 52, 18, 735944), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:52:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:52:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:25.334138+00:00 (in 5.091493 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:52:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:20 UTC)" (scheduled at 2023-01-17 05:52:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:52:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:52:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 20, 243042), True, datetime.datetime(2023, 1, 17, 5, 52, 20, 243042), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:52:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:52:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:52:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:22,626 [239] [DEBUG] [app] Starting request: urn:request:7559c2d4-27de-45b6-bcdc-148740ee8da0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,626 [243] [DEBUG] [app] Starting request: urn:request:a93a5ccd-e3ea-4b84-8013-46eddeb20568 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,628 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:22,630 [240] [DEBUG] [app] Starting request: urn:request:dccee75d-7945-4bc3-9248-9e4c48b5a8d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:22,631 [240] [DEBUG] [app] Ending request: urn:request:dccee75d-7945-4bc3-9248-9e4c48b5a8d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dccee75d-7945-4bc3-9248-9e4c48b5a8d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:22,631 [232] [DEBUG] [app] Starting request: urn:request:5be2e94b-4f06-47ae-868f-38ef418dfd52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:52:22,632 [232] [DEBUG] [app] Ending request: urn:request:5be2e94b-4f06-47ae-868f-38ef418dfd52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5be2e94b-4f06-47ae-868f-38ef418dfd52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:52:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:22,635 [244] [DEBUG] [app] Starting request: urn:request:ec046439-73c2-4c92-a435-3f32d2d106b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,636 [244] [DEBUG] [app] Ending request: urn:request:ec046439-73c2-4c92-a435-3f32d2d106b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec046439-73c2-4c92-a435-3f32d2d106b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:52:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:22,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:22,637 [244] [DEBUG] [app] Starting request: urn:request:d9232b29-01ce-4251-9f1c-f638fdb080c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,637 [244] [DEBUG] [app] Ending request: urn:request:d9232b29-01ce-4251-9f1c-f638fdb080c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9232b29-01ce-4251-9f1c-f638fdb080c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:22,647 [239] [DEBUG] [app] Ending request: urn:request:7559c2d4-27de-45b6-bcdc-148740ee8da0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7559c2d4-27de-45b6-bcdc-148740ee8da0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:22,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:52:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:52:22,648 [243] [DEBUG] [app] Ending request: urn:request:a93a5ccd-e3ea-4b84-8013-46eddeb20568 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a93a5ccd-e3ea-4b84-8013-46eddeb20568', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 05:52:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:39.230038+00:00 (in 16.392247 seconds) securityworker stdout | 2023-01-17 05:52:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:52 UTC)" (scheduled at 2023-01-17 05:52:22.837365+00:00) securityworker stdout | 2023-01-17 05:52:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:22,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:52 UTC)" executed successfully builder stdout | 2023-01-17 05:52:23,631 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:23,640 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:23,640 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:23,645 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:52:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:52:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:48.830274+00:00 (in 24.187621 seconds) exportactionlogsworker stdout | 2023-01-17 05:52:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:24 UTC)" (scheduled at 2023-01-17 05:52:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:52:24,643 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:52:24,643 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:52:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:52:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:20.242198+00:00 (in 54.907571 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:52:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:25 UTC)" (scheduled at 2023-01-17 05:52:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:52:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:52:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:52:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:52:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:57.226424+00:00 (in 29.999580 seconds) buildlogsarchiver stdout | 2023-01-17 05:52:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:57 UTC)" (scheduled at 2023-01-17 05:52:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:52:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 52, 27, 227173), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:52:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:52:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:52:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:38.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:52:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:38 UTC)" (scheduled at 2023-01-17 05:52:28.735066+00:00) notificationworker stdout | 2023-01-17 05:52:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 28, 735917), True, datetime.datetime(2023, 1, 17, 5, 52, 28, 735917), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:31,599 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:52:31,758 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:52:32,073 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:52:32,808 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:52:32,912 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:52:33,652 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:33,661 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:33,661 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:33,667 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:52:34,116 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:52:34,830 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:52:34,958 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:52:35,210 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:52:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:52:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:05.848675+00:00 (in 29.999497 seconds) gcworker stdout | 2023-01-17 05:52:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:05 UTC)" (scheduled at 2023-01-17 05:52:35.848675+00:00) gcworker stdout | 2023-01-17 05:52:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725155849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:52:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:52:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:52:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:52:36,222 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:52:36,227 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:52:36,229 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:52:36,252 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:52:36,275 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:52:37,178 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:52:37,625 [244] [DEBUG] [app] Starting request: urn:request:508653bc-ad70-4ece-9a3d-249979ef5ddd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:37,626 [239] [DEBUG] [app] Starting request: urn:request:bc9a6844-994e-4984-935b-2f4e7a5409b9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:37,630 [236] [DEBUG] [app] Starting request: urn:request:625f49f7-4e2d-4139-9ba3-cadff8696ee3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:37,630 [236] [DEBUG] [app] Ending request: urn:request:625f49f7-4e2d-4139-9ba3-cadff8696ee3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:625f49f7-4e2d-4139-9ba3-cadff8696ee3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:37,630 [232] [DEBUG] [app] Starting request: urn:request:36e8ca26-3cd3-439d-a0ba-8aaafad28094 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:37,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:52:37,631 [232] [DEBUG] [app] Ending request: urn:request:36e8ca26-3cd3-439d-a0ba-8aaafad28094 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36e8ca26-3cd3-439d-a0ba-8aaafad28094', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:37,635 [243] [DEBUG] [app] Starting request: urn:request:90e200a2-a354-484a-aa5d-946aedd9498f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:37,635 [241] [DEBUG] [app] Starting request: urn:request:c86f40a1-3198-4813-a4de-893254411669 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:37,636 [243] [DEBUG] [app] Ending request: urn:request:90e200a2-a354-484a-aa5d-946aedd9498f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90e200a2-a354-484a-aa5d-946aedd9498f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:37,636 [241] [DEBUG] [app] Ending request: urn:request:c86f40a1-3198-4813-a4de-893254411669 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c86f40a1-3198-4813-a4de-893254411669', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:52:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:37,647 [244] [DEBUG] [app] Ending request: urn:request:508653bc-ad70-4ece-9a3d-249979ef5ddd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:508653bc-ad70-4ece-9a3d-249979ef5ddd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:37,647 [239] [DEBUG] [app] Ending request: urn:request:bc9a6844-994e-4984-935b-2f4e7a5409b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc9a6844-994e-4984-935b-2f4e7a5409b9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:52:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:52:37,940 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:52:37,971 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:52:38,644 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:52:38,654 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:52:38,656 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:52:38,659 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:52:38,661 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:52:38,664 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:52:38,666 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:52:38,668 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:52:38,672 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:52:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:43.935629+00:00 (in 5.200142 seconds) notificationworker stdout | 2023-01-17 05:52:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:48 UTC)" (scheduled at 2023-01-17 05:52:38.735066+00:00) notificationworker stdout | 2023-01-17 05:52:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 38, 735843), True, datetime.datetime(2023, 1, 17, 5, 52, 38, 735843), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:38,761 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:52:38,764 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:52:38,766 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:52:38,769 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:52:38,771 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:52:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:52:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:48.738458+00:00 (in 9.707159 seconds) repositorygcworker stdout | 2023-01-17 05:52:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:39 UTC)" (scheduled at 2023-01-17 05:52:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:52:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:52:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 39, 31726), True, datetime.datetime(2023, 1, 17, 5, 52, 39, 31726), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:52:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:52:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:52:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:52:39,156 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:52:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:52.837365+00:00 (in 13.606859 seconds) securityworker stdout | 2023-01-17 05:52:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:09 UTC)" (scheduled at 2023-01-17 05:52:39.230038+00:00) securityworker stdout | 2023-01-17 05:52:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:52:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:52:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:52:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:39,727 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:52:40,490 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 05:52:40,524 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 05:52:40,539 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 05:52:40,684 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:52:43,677 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:43,686 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:43,686 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:43,692 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:48.735066+00:00 (in 4.798964 seconds) notificationworker stdout | 2023-01-17 05:52:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:43 UTC)" (scheduled at 2023-01-17 05:52:43.935629+00:00) notificationworker stdout | 2023-01-17 05:52:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:52:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:43 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:52:45,048 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 05:52:45,048 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:52:45.046803+00:00 (in 3599.998497 seconds) blobuploadcleanupworker stdout | 2023-01-17 05:52:45,048 [59] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2023-01-17 06:52:45 UTC)" (scheduled at 2023-01-17 05:52:45.046803+00:00) blobuploadcleanupworker stdout | 2023-01-17 05:52:45,048 [59] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:52:45,048 [59] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ... blobuploadcleanupworker stdout | 2023-01-17 05:52:45,049 [59] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2023-01-17 05:52:45,049 [59] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:52:45,049 [59] [DEBUG] [__main__] Performing blob upload cleanup blobuploadcleanupworker stdout | 2023-01-17 05:52:45,049 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 15, 5, 52, 45, 49217), 1, 0]) blobuploadcleanupworker stdout | 2023-01-17 05:52:45,058 [59] [DEBUG] [__main__] No additional stale blob uploads found blobuploadcleanupworker stdout | 2023-01-17 05:52:45,058 [59] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 05:52:45,058 [59] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:52:45,059 [59] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2023-01-17 05:52:45,059 [59] [DEBUG] [util.locking] Released lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:52:45,059 [59] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2023-01-17 06:52:45 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:52:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:52:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:59.337413+00:00 (in 11.404493 seconds) namespacegcworker stdout | 2023-01-17 05:52:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:47 UTC)" (scheduled at 2023-01-17 05:52:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:52:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:52:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:58.735066+00:00 (in 9.999437 seconds) notificationworker stdout | 2023-01-17 05:52:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:58 UTC)" (scheduled at 2023-01-17 05:52:48.735066+00:00) notificationworker stdout | 2023-01-17 05:52:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 48, 736045), True, datetime.datetime(2023, 1, 17, 5, 52, 48, 736045), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:52:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:52:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:39.030814+00:00 (in 50.291884 seconds) repositorygcworker stdout | 2023-01-17 05:52:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:48 UTC)" (scheduled at 2023-01-17 05:52:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:52:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:52:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:48,750 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:48,750 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:48,751 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:52:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:52:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:24.641816+00:00 (in 35.811110 seconds) exportactionlogsworker stdout | 2023-01-17 05:52:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:48 UTC)" (scheduled at 2023-01-17 05:52:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:52:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:52:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 48, 831115), True, datetime.datetime(2023, 1, 17, 5, 52, 48, 831115), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:52:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:52:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:52:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:52,625 [241] [DEBUG] [app] Starting request: urn:request:c48bce45-d378-425c-867c-4290dd135d9f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:52,625 [239] [DEBUG] [app] Starting request: urn:request:0d8b721b-ebfe-41cf-9031-4ccabfef956b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:52:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:52,630 [232] [DEBUG] [app] Starting request: urn:request:955e970f-b5ed-44ff-9ed2-14ed1ee14dcf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,630 [242] [DEBUG] [app] Starting request: urn:request:1b025d88-07d6-4398-82b1-7c9ed1260687 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,631 [232] [DEBUG] [app] Ending request: urn:request:955e970f-b5ed-44ff-9ed2-14ed1ee14dcf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:955e970f-b5ed-44ff-9ed2-14ed1ee14dcf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,631 [242] [DEBUG] [app] Ending request: urn:request:1b025d88-07d6-4398-82b1-7c9ed1260687 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b025d88-07d6-4398-82b1-7c9ed1260687', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:52:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:52,636 [243] [DEBUG] [app] Starting request: urn:request:c6b11736-e657-4848-99e9-469185815f51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:52,636 [244] [DEBUG] [app] Starting request: urn:request:490ceccd-8883-4ec2-9c94-bb756005055b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:52,636 [243] [DEBUG] [app] Ending request: urn:request:c6b11736-e657-4848-99e9-469185815f51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6b11736-e657-4848-99e9-469185815f51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:52,636 [244] [DEBUG] [app] Ending request: urn:request:490ceccd-8883-4ec2-9c94-bb756005055b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:490ceccd-8883-4ec2-9c94-bb756005055b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:52,648 [239] [DEBUG] [app] Ending request: urn:request:0d8b721b-ebfe-41cf-9031-4ccabfef956b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d8b721b-ebfe-41cf-9031-4ccabfef956b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:52,648 [241] [DEBUG] [app] Ending request: urn:request:c48bce45-d378-425c-867c-4290dd135d9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c48bce45-d378-425c-867c-4290dd135d9f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 05:52:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:52:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:09.230038+00:00 (in 16.392227 seconds) securityworker stdout | 2023-01-17 05:52:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:22 UTC)" (scheduled at 2023-01-17 05:52:52.837365+00:00) securityworker stdout | 2023-01-17 05:52:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:22 UTC)" executed successfully builder stdout | 2023-01-17 05:52:53,695 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:53,703 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:53,703 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:53,709 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:52:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:52:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:27.226424+00:00 (in 29.999567 seconds) buildlogsarchiver stdout | 2023-01-17 05:52:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:27 UTC)" (scheduled at 2023-01-17 05:52:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:52:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 52, 57, 227232), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:52:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:52:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:52:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:08.735066+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 05:52:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:08 UTC)" (scheduled at 2023-01-17 05:52:58.735066+00:00) notificationworker stdout | 2023-01-17 05:52:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 58, 735982), True, datetime.datetime(2023, 1, 17, 5, 52, 58, 735982), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:52:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:52:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:47.932490+00:00 (in 48.594624 seconds) namespacegcworker stdout | 2023-01-17 05:52:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:59 UTC)" (scheduled at 2023-01-17 05:52:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:52:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:52:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 59, 338349), True, datetime.datetime(2023, 1, 17, 5, 52, 59, 338349), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:52:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:52:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:52:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:01,604 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:53:01,767 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:53:02,108 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:53:02,844 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:53:02,930 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:53:03,719 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:03,728 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:03,728 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:03,734 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:53:04,145 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:53:04,847 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:53:04,993 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:53:05,240 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:53:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:53:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:35.848675+00:00 (in 29.999551 seconds) gcworker stdout | 2023-01-17 05:53:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:35 UTC)" (scheduled at 2023-01-17 05:53:05.848675+00:00) gcworker stdout | 2023-01-17 05:53:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725185849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:53:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:53:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:53:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:53:06,229 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:53:06,232 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:53:06,236 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:53:06,279 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:53:06,293 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:53:07,184 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:53:07,625 [243] [DEBUG] [app] Starting request: urn:request:5bb9d5fc-0ae2-42ab-868f-48700492896b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:07,625 [239] [DEBUG] [app] Starting request: urn:request:4f97eaeb-a7e0-43ca-a918-8732206aeb62 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:07,630 [242] [DEBUG] [app] Starting request: urn:request:e589d9a3-d699-4397-93d1-380e764f08d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:07,630 [236] [DEBUG] [app] Starting request: urn:request:720d1c52-45b1-4502-ade4-a87ca9da6de8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:07,631 [242] [DEBUG] [app] Ending request: urn:request:e589d9a3-d699-4397-93d1-380e764f08d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e589d9a3-d699-4397-93d1-380e764f08d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:53:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:53:07,631 [236] [DEBUG] [app] Ending request: urn:request:720d1c52-45b1-4502-ade4-a87ca9da6de8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:720d1c52-45b1-4502-ade4-a87ca9da6de8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:53:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:53:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:07,635 [244] [DEBUG] [app] Starting request: urn:request:07e0e307-c8e6-4a35-ab51-9bab4641397d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:07,636 [244] [DEBUG] [app] Ending request: urn:request:07e0e307-c8e6-4a35-ab51-9bab4641397d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:07e0e307-c8e6-4a35-ab51-9bab4641397d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:07,636 [241] [DEBUG] [app] Starting request: urn:request:dd783cdf-6cd4-419f-b2ff-f125de96879a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:07,636 [241] [DEBUG] [app] Ending request: urn:request:dd783cdf-6cd4-419f-b2ff-f125de96879a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd783cdf-6cd4-419f-b2ff-f125de96879a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:07,646 [243] [DEBUG] [app] Ending request: urn:request:5bb9d5fc-0ae2-42ab-868f-48700492896b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5bb9d5fc-0ae2-42ab-868f-48700492896b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:07,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:53:07,647 [239] [DEBUG] [app] Ending request: urn:request:4f97eaeb-a7e0-43ca-a918-8732206aeb62 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f97eaeb-a7e0-43ca-a918-8732206aeb62', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) logrotateworker stdout | 2023-01-17 05:53:07,976 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:53:07,986 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:53:08,651 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:53:08,661 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:53:08,667 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:53:08,670 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:53:08,672 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:53:08,674 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:53:08,677 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:53:08,680 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:53:08,682 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:53:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:18.735066+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:53:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:18 UTC)" (scheduled at 2023-01-17 05:53:08.735066+00:00) notificationworker stdout | 2023-01-17 05:53:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 8, 735878), True, datetime.datetime(2023, 1, 17, 5, 53, 8, 735878), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:53:08,767 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:53:08,771 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:53:08,773 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:53:08,776 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:53:08,778 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:53:09,183 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:53:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:22.837365+00:00 (in 13.606899 seconds) securityworker stdout | 2023-01-17 05:53:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:39 UTC)" (scheduled at 2023-01-17 05:53:09.230038+00:00) securityworker stdout | 2023-01-17 05:53:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:53:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:53:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:53:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:09,763 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:53:10,517 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:53:10,554 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:53:10,556 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:53:10,718 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:53:13,744 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:13,753 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:13,753 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:13,759 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:28.735066+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:53:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:28 UTC)" (scheduled at 2023-01-17 05:53:18.735066+00:00) notificationworker stdout | 2023-01-17 05:53:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 18, 735873), True, datetime.datetime(2023, 1, 17, 5, 53, 18, 735873), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:53:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:53:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:25.334138+00:00 (in 5.091441 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:53:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:20 UTC)" (scheduled at 2023-01-17 05:53:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:53:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:53:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 20, 243090), True, datetime.datetime(2023, 1, 17, 5, 53, 20, 243090), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:53:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:53:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:53:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:53:22,625 [239] [DEBUG] [app] Starting request: urn:request:5c0a30ac-1338-4950-a237-1704b6f44f4b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:22,625 [244] [DEBUG] [app] Starting request: urn:request:4704311c-fe81-463e-b0c1-074e109de87f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:22,630 [237] [DEBUG] [app] Starting request: urn:request:e419888b-41ff-44f7-a931-9da4be879d7e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,631 [236] [DEBUG] [app] Starting request: urn:request:c577dfdc-a500-4853-b2c9-5aeb4972638e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,631 [237] [DEBUG] [app] Ending request: urn:request:e419888b-41ff-44f7-a931-9da4be879d7e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e419888b-41ff-44f7-a931-9da4be879d7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:53:22,631 [236] [DEBUG] [app] Ending request: urn:request:c577dfdc-a500-4853-b2c9-5aeb4972638e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c577dfdc-a500-4853-b2c9-5aeb4972638e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:53:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:53:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:22,635 [243] [DEBUG] [app] Starting request: urn:request:5d657fdf-7ae5-457f-b90a-988079ed9833 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:22,636 [243] [DEBUG] [app] Ending request: urn:request:5d657fdf-7ae5-457f-b90a-988079ed9833 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d657fdf-7ae5-457f-b90a-988079ed9833', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:22,636 [241] [DEBUG] [app] Starting request: urn:request:aaf087a7-1ca5-40e0-b74e-585f9bf6ce41 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:53:22,636 [241] [DEBUG] [app] Ending request: urn:request:aaf087a7-1ca5-40e0-b74e-585f9bf6ce41 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aaf087a7-1ca5-40e0-b74e-585f9bf6ce41', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:22,647 [239] [DEBUG] [app] Ending request: urn:request:5c0a30ac-1338-4950-a237-1704b6f44f4b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5c0a30ac-1338-4950-a237-1704b6f44f4b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:22,647 [244] [DEBUG] [app] Ending request: urn:request:4704311c-fe81-463e-b0c1-074e109de87f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4704311c-fe81-463e-b0c1-074e109de87f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:53:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:53:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:39.230038+00:00 (in 16.392230 seconds) securityworker stdout | 2023-01-17 05:53:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:52 UTC)" (scheduled at 2023-01-17 05:53:22.837365+00:00) securityworker stdout | 2023-01-17 05:53:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:52 UTC)" executed successfully builder stdout | 2023-01-17 05:53:23,769 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:23,777 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:23,777 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:23,783 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:53:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:53:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:48.830274+00:00 (in 24.187991 seconds) exportactionlogsworker stdout | 2023-01-17 05:53:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:24 UTC)" (scheduled at 2023-01-17 05:53:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:53:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:53:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:53:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:53:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:20.242198+00:00 (in 54.907569 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:53:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:25 UTC)" (scheduled at 2023-01-17 05:53:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:53:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:53:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:53:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:53:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:57.226424+00:00 (in 29.999551 seconds) buildlogsarchiver stdout | 2023-01-17 05:53:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:57 UTC)" (scheduled at 2023-01-17 05:53:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:53:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 53, 27, 227230), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:53:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:53:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:53:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:28,736 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:38.735066+00:00 (in 9.999139 seconds) notificationworker stdout | 2023-01-17 05:53:28,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:38 UTC)" (scheduled at 2023-01-17 05:53:28.735066+00:00) notificationworker stdout | 2023-01-17 05:53:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 28, 736342), True, datetime.datetime(2023, 1, 17, 5, 53, 28, 736342), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:31,622 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:53:31,803 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:53:32,128 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:53:32,876 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:53:32,948 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:53:33,788 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:33,797 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:33,797 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:33,803 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:53:34,181 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:53:34,879 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:53:34,998 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:53:35,276 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:53:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:53:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:05.848675+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 05:53:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:05 UTC)" (scheduled at 2023-01-17 05:53:35.848675+00:00) gcworker stdout | 2023-01-17 05:53:35,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:53:35,856 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725215856, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:53:35,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:53:35,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:53:35,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:53:36,237 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:53:36,239 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:53:36,242 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:53:36,302 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:53:36,313 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:53:37,197 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:53:37,625 [239] [DEBUG] [app] Starting request: urn:request:a18cf8b1-5bf7-4444-8174-ef8cb825c503 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:37,625 [241] [DEBUG] [app] Starting request: urn:request:d9619603-7559-4302-82ec-dc4a0bc50d07 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:37,630 [232] [DEBUG] [app] Starting request: urn:request:10be44aa-e7aa-4c8f-aa49-080fcd33b24f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:37,630 [242] [DEBUG] [app] Starting request: urn:request:8727a1f6-d260-4091-92cb-87438733b64f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:37,630 [232] [DEBUG] [app] Ending request: urn:request:10be44aa-e7aa-4c8f-aa49-080fcd33b24f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:10be44aa-e7aa-4c8f-aa49-080fcd33b24f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:53:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:53:37,631 [242] [DEBUG] [app] Ending request: urn:request:8727a1f6-d260-4091-92cb-87438733b64f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8727a1f6-d260-4091-92cb-87438733b64f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:53:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:53:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:37,635 [243] [DEBUG] [app] Starting request: urn:request:fb457288-2761-4f45-82bd-44bf3fc40d3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:37,636 [239] [DEBUG] [app] Starting request: urn:request:abf55e74-bcf2-4655-b2f8-d2e29d0e8124 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:37,636 [243] [DEBUG] [app] Ending request: urn:request:fb457288-2761-4f45-82bd-44bf3fc40d3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb457288-2761-4f45-82bd-44bf3fc40d3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:37,636 [239] [DEBUG] [app] Ending request: urn:request:abf55e74-bcf2-4655-b2f8-d2e29d0e8124 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abf55e74-bcf2-4655-b2f8-d2e29d0e8124', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:53:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:37,647 [241] [DEBUG] [app] Ending request: urn:request:d9619603-7559-4302-82ec-dc4a0bc50d07 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9619603-7559-4302-82ec-dc4a0bc50d07', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:37,647 [239] [DEBUG] [app] Ending request: urn:request:a18cf8b1-5bf7-4444-8174-ef8cb825c503 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a18cf8b1-5bf7-4444-8174-ef8cb825c503', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:53:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:53:38,006 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:53:38,021 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:53:38,662 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:53:38,667 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:53:38,676 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:53:38,678 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:53:38,681 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:53:38,685 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:53:38,687 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:53:38,689 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:53:38,692 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:53:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:43.935629+00:00 (in 5.200114 seconds) notificationworker stdout | 2023-01-17 05:53:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:48 UTC)" (scheduled at 2023-01-17 05:53:38.735066+00:00) notificationworker stdout | 2023-01-17 05:53:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 38, 735896), True, datetime.datetime(2023, 1, 17, 5, 53, 38, 735896), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:53:38,774 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:53:38,777 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:53:38,780 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:53:38,783 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:53:38,785 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:53:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:53:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:48.738458+00:00 (in 9.707172 seconds) repositorygcworker stdout | 2023-01-17 05:53:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:39 UTC)" (scheduled at 2023-01-17 05:53:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:53:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:53:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 39, 31755), True, datetime.datetime(2023, 1, 17, 5, 53, 39, 31755), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:53:39,041 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:53:39,041 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:53:39,041 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:53:39,190 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:53:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:52.837365+00:00 (in 13.606892 seconds) securityworker stdout | 2023-01-17 05:53:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:09 UTC)" (scheduled at 2023-01-17 05:53:39.230038+00:00) securityworker stdout | 2023-01-17 05:53:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:53:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:53:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:53:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:53:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:53:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:53:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:39,771 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:53:40,551 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:53:40,585 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:53:40,591 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:53:40,740 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:53:43,813 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:43,822 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:43,822 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:43,828 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:48.735066+00:00 (in 4.798972 seconds) notificationworker stdout | 2023-01-17 05:53:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:43 UTC)" (scheduled at 2023-01-17 05:53:43.935629+00:00) notificationworker stdout | 2023-01-17 05:53:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:53:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:53:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:53:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:59.337413+00:00 (in 11.404453 seconds) namespacegcworker stdout | 2023-01-17 05:53:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:47 UTC)" (scheduled at 2023-01-17 05:53:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:53:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:53:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:56.935055+00:00 (in 8.199527 seconds) notificationworker stdout | 2023-01-17 05:53:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:58 UTC)" (scheduled at 2023-01-17 05:53:48.735066+00:00) notificationworker stdout | 2023-01-17 05:53:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 48, 735914), True, datetime.datetime(2023, 1, 17, 5, 53, 48, 735914), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:53:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:53:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:39.030814+00:00 (in 50.291859 seconds) repositorygcworker stdout | 2023-01-17 05:53:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:48 UTC)" (scheduled at 2023-01-17 05:53:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:53:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:53:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:53:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:53:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:24.641816+00:00 (in 35.811073 seconds) exportactionlogsworker stdout | 2023-01-17 05:53:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:48 UTC)" (scheduled at 2023-01-17 05:53:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:53:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:53:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 48, 831107), True, datetime.datetime(2023, 1, 17, 5, 53, 48, 831107), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:53:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:53:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:53:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:53:52,625 [239] [DEBUG] [app] Starting request: urn:request:1005b9ca-20fb-48e7-9765-051aabc993b7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:52,625 [243] [DEBUG] [app] Starting request: urn:request:8335507c-9935-4823-9db7-3ec9d264792c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:53:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:52,630 [232] [DEBUG] [app] Starting request: urn:request:36f38f6e-679b-4fdb-b19b-0b9242fd5d34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:52,630 [242] [DEBUG] [app] Starting request: urn:request:e227d79a-75f1-41d5-8c37-065cd20d5629 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:52,630 [232] [DEBUG] [app] Ending request: urn:request:36f38f6e-679b-4fdb-b19b-0b9242fd5d34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36f38f6e-679b-4fdb-b19b-0b9242fd5d34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:53:52,630 [242] [DEBUG] [app] Ending request: urn:request:e227d79a-75f1-41d5-8c37-065cd20d5629 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e227d79a-75f1-41d5-8c37-065cd20d5629', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:53:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:53:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:52,635 [241] [DEBUG] [app] Starting request: urn:request:a9d2ab4b-3fe5-4733-b1b1-6c8b65cd68eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:52,635 [244] [DEBUG] [app] Starting request: urn:request:9df273c4-7800-4537-91ee-3cdd84196d68 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:52,636 [241] [DEBUG] [app] Ending request: urn:request:a9d2ab4b-3fe5-4733-b1b1-6c8b65cd68eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9d2ab4b-3fe5-4733-b1b1-6c8b65cd68eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:52,636 [244] [DEBUG] [app] Ending request: urn:request:9df273c4-7800-4537-91ee-3cdd84196d68 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9df273c4-7800-4537-91ee-3cdd84196d68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:53:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:52,646 [243] [DEBUG] [app] Ending request: urn:request:8335507c-9935-4823-9db7-3ec9d264792c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8335507c-9935-4823-9db7-3ec9d264792c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:52,647 [239] [DEBUG] [app] Ending request: urn:request:1005b9ca-20fb-48e7-9765-051aabc993b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1005b9ca-20fb-48e7-9765-051aabc993b7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:53:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:53:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:09.230038+00:00 (in 16.392227 seconds) securityworker stdout | 2023-01-17 05:53:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:22 UTC)" (scheduled at 2023-01-17 05:53:52.837365+00:00) securityworker stdout | 2023-01-17 05:53:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:22 UTC)" executed successfully builder stdout | 2023-01-17 05:53:53,828 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:53,836 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:53,836 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:53,842 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:58.735066+00:00 (in 1.799547 seconds) notificationworker stdout | 2023-01-17 05:53:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:58:56 UTC)" (scheduled at 2023-01-17 05:53:56.935055+00:00) notificationworker stdout | 2023-01-17 05:53:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 53, 56, 935903), 'notification/%']) notificationworker stdout | 2023-01-17 05:53:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 53, 56, 935903), True, datetime.datetime(2023, 1, 17, 5, 53, 56, 935903), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:53:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 53, 56, 935903), True, datetime.datetime(2023, 1, 17, 5, 53, 56, 935903), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 53, 56, 935903), 'notification/%']) notificationworker stdout | 2023-01-17 05:53:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:58:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:53:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:53:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:27.226424+00:00 (in 29.999580 seconds) buildlogsarchiver stdout | 2023-01-17 05:53:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:27 UTC)" (scheduled at 2023-01-17 05:53:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:53:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 53, 57, 227186), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:53:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:53:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:53:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:08.735066+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:53:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:08 UTC)" (scheduled at 2023-01-17 05:53:58.735066+00:00) notificationworker stdout | 2023-01-17 05:53:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 58, 735936), True, datetime.datetime(2023, 1, 17, 5, 53, 58, 735936), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:53:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:53:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:15.931994+00:00 (in 16.594112 seconds) namespacegcworker stdout | 2023-01-17 05:53:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:59 UTC)" (scheduled at 2023-01-17 05:53:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:53:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:53:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 59, 338743), True, datetime.datetime(2023, 1, 17, 5, 53, 59, 338743), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:53:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:53:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:53:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:01,658 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:54:01,819 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:54:02,133 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:54:02,893 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:54:02,983 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:54:03,850 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:03,859 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:03,859 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:03,865 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:54:04,205 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:54:04,909 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:54:05,034 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:54:05,307 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:54:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:54:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:35.848675+00:00 (in 29.999494 seconds) gcworker stdout | 2023-01-17 05:54:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:35 UTC)" (scheduled at 2023-01-17 05:54:05.848675+00:00) gcworker stdout | 2023-01-17 05:54:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725245849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:54:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:54:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:54:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:54:06,245 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:54:06,247 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:54:06,250 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:54:06,338 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:54:06,341 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:54:07,230 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:54:07,625 [244] [DEBUG] [app] Starting request: urn:request:fdb43757-69b6-4d56-a6b2-23d108123ae8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:07,625 [239] [DEBUG] [app] Starting request: urn:request:61d0b6ab-3d7d-4745-85ee-adb4502c71e0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:07,630 [232] [DEBUG] [app] Starting request: urn:request:d43bafce-6d85-49bd-a84b-64b993366c0b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:07,630 [242] [DEBUG] [app] Starting request: urn:request:50d2c795-663e-4055-b73e-2a83931f4422 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:07,630 [232] [DEBUG] [app] Ending request: urn:request:d43bafce-6d85-49bd-a84b-64b993366c0b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d43bafce-6d85-49bd-a84b-64b993366c0b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:54:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:54:07,630 [242] [DEBUG] [app] Ending request: urn:request:50d2c795-663e-4055-b73e-2a83931f4422 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50d2c795-663e-4055-b73e-2a83931f4422', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:54:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:07,635 [243] [DEBUG] [app] Starting request: urn:request:7586f354-0b75-4756-ada3-3d2e19e213d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:07,635 [239] [DEBUG] [app] Starting request: urn:request:414d497c-ac2e-4f2d-8e0c-700939a7b624 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:07,636 [243] [DEBUG] [app] Ending request: urn:request:7586f354-0b75-4756-ada3-3d2e19e213d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7586f354-0b75-4756-ada3-3d2e19e213d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:07,636 [239] [DEBUG] [app] Ending request: urn:request:414d497c-ac2e-4f2d-8e0c-700939a7b624 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:414d497c-ac2e-4f2d-8e0c-700939a7b624', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:07,646 [239] [DEBUG] [app] Ending request: urn:request:61d0b6ab-3d7d-4745-85ee-adb4502c71e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61d0b6ab-3d7d-4745-85ee-adb4502c71e0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:07,647 [244] [DEBUG] [app] Ending request: urn:request:fdb43757-69b6-4d56-a6b2-23d108123ae8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fdb43757-69b6-4d56-a6b2-23d108123ae8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:54:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) logrotateworker stdout | 2023-01-17 05:54:08,029 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:54:08,031 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:54:08,673 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:54:08,675 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:54:08,682 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:54:08,686 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:54:08,691 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:54:08,694 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:54:08,696 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:54:08,698 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:54:08,701 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:54:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:18.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 05:54:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:18 UTC)" (scheduled at 2023-01-17 05:54:08.735066+00:00) notificationworker stdout | 2023-01-17 05:54:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 8, 735906), True, datetime.datetime(2023, 1, 17, 5, 54, 8, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:54:08,781 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:54:08,785 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:54:08,787 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:54:08,789 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:54:08,792 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:54:09,199 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:54:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:22.837365+00:00 (in 13.606912 seconds) securityworker stdout | 2023-01-17 05:54:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:39 UTC)" (scheduled at 2023-01-17 05:54:09.230038+00:00) securityworker stdout | 2023-01-17 05:54:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:54:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:54:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:54:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:09,807 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:54:10,574 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:54:10,620 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:54:10,628 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:54:10,774 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:54:13,875 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:13,884 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:13,884 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:13,890 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:54:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:54:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:47.932490+00:00 (in 31.999964 seconds) namespacegcworker stdout | 2023-01-17 05:54:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:59:15 UTC)" (scheduled at 2023-01-17 05:54:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:54:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 54, 15, 932984), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:54:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 54, 15, 932984), True, datetime.datetime(2023, 1, 17, 5, 54, 15, 932984), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:54:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 54, 15, 932984), True, datetime.datetime(2023, 1, 17, 5, 54, 15, 932984), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 54, 15, 932984), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:54:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:54:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:59:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:28.735066+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 05:54:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:28 UTC)" (scheduled at 2023-01-17 05:54:18.735066+00:00) notificationworker stdout | 2023-01-17 05:54:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 18, 735919), True, datetime.datetime(2023, 1, 17, 5, 54, 18, 735919), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:54:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:54:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:25.334138+00:00 (in 5.091460 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:54:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:20 UTC)" (scheduled at 2023-01-17 05:54:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:54:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:54:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 20, 243160), True, datetime.datetime(2023, 1, 17, 5, 54, 20, 243160), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:54:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:54:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:54:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:54:22,626 [239] [DEBUG] [app] Starting request: urn:request:23326739-68c8-4f22-b097-a4c784bccc2f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:22,626 [244] [DEBUG] [app] Starting request: urn:request:2fe0cf43-d3b2-4c53-9d82-b1898a2e8ac3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:22,630 [240] [DEBUG] [app] Starting request: urn:request:bef27f03-af0e-4dfa-9d20-66c99bb163e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:22,631 [232] [DEBUG] [app] Starting request: urn:request:14bad8f4-abdf-4441-b5f4-66ccd35d42b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:22,631 [240] [DEBUG] [app] Ending request: urn:request:bef27f03-af0e-4dfa-9d20-66c99bb163e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bef27f03-af0e-4dfa-9d20-66c99bb163e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:54:22,631 [232] [DEBUG] [app] Ending request: urn:request:14bad8f4-abdf-4441-b5f4-66ccd35d42b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:14bad8f4-abdf-4441-b5f4-66ccd35d42b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:54:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:54:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:22,636 [243] [DEBUG] [app] Starting request: urn:request:faf82293-24f2-4817-ac33-5720e4c18fd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:22,636 [239] [DEBUG] [app] Starting request: urn:request:0c8f0288-7efc-4eec-a782-103f53843308 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:22,636 [243] [DEBUG] [app] Ending request: urn:request:faf82293-24f2-4817-ac33-5720e4c18fd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:faf82293-24f2-4817-ac33-5720e4c18fd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:22,637 [239] [DEBUG] [app] Ending request: urn:request:0c8f0288-7efc-4eec-a782-103f53843308 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c8f0288-7efc-4eec-a782-103f53843308', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:22,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:22,648 [239] [DEBUG] [app] Ending request: urn:request:23326739-68c8-4f22-b097-a4c784bccc2f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23326739-68c8-4f22-b097-a4c784bccc2f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:22,648 [244] [DEBUG] [app] Ending request: urn:request:2fe0cf43-d3b2-4c53-9d82-b1898a2e8ac3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2fe0cf43-d3b2-4c53-9d82-b1898a2e8ac3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:54:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:54:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:39.230038+00:00 (in 16.392275 seconds) securityworker stdout | 2023-01-17 05:54:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:52 UTC)" (scheduled at 2023-01-17 05:54:22.837365+00:00) securityworker stdout | 2023-01-17 05:54:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:52 UTC)" executed successfully builder stdout | 2023-01-17 05:54:23,900 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:23,909 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:23,909 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:23,914 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:54:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:54:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:48.830274+00:00 (in 24.188008 seconds) exportactionlogsworker stdout | 2023-01-17 05:54:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:24 UTC)" (scheduled at 2023-01-17 05:54:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:54:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:54:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:54:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:54:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:20.242198+00:00 (in 54.907495 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:54:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:25 UTC)" (scheduled at 2023-01-17 05:54:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:54:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:54:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:54:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:54:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:57.226424+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 05:54:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:57 UTC)" (scheduled at 2023-01-17 05:54:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:54:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 54, 27, 227274), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:54:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:54:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:54:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:38.735066+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:54:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:38 UTC)" (scheduled at 2023-01-17 05:54:28.735066+00:00) notificationworker stdout | 2023-01-17 05:54:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 28, 735977), True, datetime.datetime(2023, 1, 17, 5, 54, 28, 735977), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:31,669 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:54:31,855 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:54:32,169 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:54:32,913 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:54:33,018 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:54:33,924 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:33,934 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:33,934 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:33,939 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:54:34,242 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:54:34,915 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:54:35,039 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:54:35,342 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:54:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:54:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:05.848675+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 05:54:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:05 UTC)" (scheduled at 2023-01-17 05:54:35.848675+00:00) gcworker stdout | 2023-01-17 05:54:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725275849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:54:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:54:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:54:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:54:36,252 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:54:36,254 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:54:36,256 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:54:36,353 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:54:36,367 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:54:37,261 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:54:37,625 [239] [DEBUG] [app] Starting request: urn:request:c0e800bd-2c92-4a5f-9072-ffd308b58a2c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:37,625 [244] [DEBUG] [app] Starting request: urn:request:8996053d-e310-4b57-9c98-6e69b2c53cab (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:37,630 [242] [DEBUG] [app] Starting request: urn:request:a5e9484c-1f9d-4b72-b740-8304fb5f78c4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,631 [235] [DEBUG] [app] Starting request: urn:request:bd3106ff-9b8f-4c07-86d9-4c2273174a4f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,631 [242] [DEBUG] [app] Ending request: urn:request:a5e9484c-1f9d-4b72-b740-8304fb5f78c4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5e9484c-1f9d-4b72-b740-8304fb5f78c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:54:37,631 [235] [DEBUG] [app] Ending request: urn:request:bd3106ff-9b8f-4c07-86d9-4c2273174a4f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd3106ff-9b8f-4c07-86d9-4c2273174a4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:54:37,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:37,636 [244] [DEBUG] [app] Starting request: urn:request:51edafb2-9977-4dce-a789-797e88e94a50 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:37,636 [244] [DEBUG] [app] Ending request: urn:request:51edafb2-9977-4dce-a789-797e88e94a50 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:51edafb2-9977-4dce-a789-797e88e94a50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:37,636 [243] [DEBUG] [app] Starting request: urn:request:05e0adbb-dfad-452a-a879-29eb92392786 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:37,637 [243] [DEBUG] [app] Ending request: urn:request:05e0adbb-dfad-452a-a879-29eb92392786 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05e0adbb-dfad-452a-a879-29eb92392786', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:54:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:37,647 [239] [DEBUG] [app] Ending request: urn:request:c0e800bd-2c92-4a5f-9072-ffd308b58a2c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0e800bd-2c92-4a5f-9072-ffd308b58a2c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:37,647 [244] [DEBUG] [app] Ending request: urn:request:8996053d-e310-4b57-9c98-6e69b2c53cab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8996053d-e310-4b57-9c98-6e69b2c53cab', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:54:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) logrotateworker stdout | 2023-01-17 05:54:38,057 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:54:38,066 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:54:38,682 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:54:38,684 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:54:38,692 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:54:38,694 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:54:38,700 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:54:38,704 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:54:38,706 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:54:38,708 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:54:38,711 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:54:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:43.935629+00:00 (in 5.200112 seconds) notificationworker stdout | 2023-01-17 05:54:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:48 UTC)" (scheduled at 2023-01-17 05:54:38.735066+00:00) notificationworker stdout | 2023-01-17 05:54:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 38, 735907), True, datetime.datetime(2023, 1, 17, 5, 54, 38, 735907), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:54:38,788 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:54:38,791 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:54:38,794 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:54:38,796 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:54:38,798 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:54:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:54:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:48.738458+00:00 (in 9.707160 seconds) repositorygcworker stdout | 2023-01-17 05:54:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:39 UTC)" (scheduled at 2023-01-17 05:54:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:54:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:54:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 39, 31751), True, datetime.datetime(2023, 1, 17, 5, 54, 39, 31751), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:54:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:54:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:54:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:54:39,205 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:54:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:52.837365+00:00 (in 13.606888 seconds) securityworker stdout | 2023-01-17 05:54:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:09 UTC)" (scheduled at 2023-01-17 05:54:39.230038+00:00) securityworker stdout | 2023-01-17 05:54:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:54:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:54:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:54:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:54:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:54:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:54:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:09 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:39,824 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:54:40,587 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:54:40,626 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:54:40,662 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:54:40,800 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:54:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:48.735066+00:00 (in 4.798972 seconds) notificationworker stdout | 2023-01-17 05:54:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:43 UTC)" (scheduled at 2023-01-17 05:54:43.935629+00:00) notificationworker stdout | 2023-01-17 05:54:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:54:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:43 UTC)" executed successfully builder stdout | 2023-01-17 05:54:43,950 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:43,959 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:43,959 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:43,965 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:54:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:54:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:59.337413+00:00 (in 11.404385 seconds) namespacegcworker stdout | 2023-01-17 05:54:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:47 UTC)" (scheduled at 2023-01-17 05:54:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:54:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:54:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:58.735066+00:00 (in 9.999485 seconds) notificationworker stdout | 2023-01-17 05:54:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:58 UTC)" (scheduled at 2023-01-17 05:54:48.735066+00:00) notificationworker stdout | 2023-01-17 05:54:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:48,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 48, 736040), True, datetime.datetime(2023, 1, 17, 5, 54, 48, 736040), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:54:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:54:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:39.030814+00:00 (in 50.291884 seconds) repositorygcworker stdout | 2023-01-17 05:54:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:48 UTC)" (scheduled at 2023-01-17 05:54:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:54:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:54:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:48,750 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:48,750 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:48,750 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:54:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:54:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:24.641816+00:00 (in 35.811062 seconds) exportactionlogsworker stdout | 2023-01-17 05:54:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:48 UTC)" (scheduled at 2023-01-17 05:54:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:54:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:54:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 48, 831112), True, datetime.datetime(2023, 1, 17, 5, 54, 48, 831112), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:54:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:54:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:54:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:54:52,625 [243] [DEBUG] [app] Starting request: urn:request:42aac001-dedc-48bb-873e-cb602df35988 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:52,625 [239] [DEBUG] [app] Starting request: urn:request:0662155f-8a2e-49b5-a9b0-b511f6de9110 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:54:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:52,630 [232] [DEBUG] [app] Starting request: urn:request:cc40b037-eec5-41da-971c-0600a714e226 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:52,630 [240] [DEBUG] [app] Starting request: urn:request:0a985099-4938-4c6b-abef-128e1705d074 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:52,630 [232] [DEBUG] [app] Ending request: urn:request:cc40b037-eec5-41da-971c-0600a714e226 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc40b037-eec5-41da-971c-0600a714e226', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:54:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:54:52,631 [240] [DEBUG] [app] Ending request: urn:request:0a985099-4938-4c6b-abef-128e1705d074 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a985099-4938-4c6b-abef-128e1705d074', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:54:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:52,635 [244] [DEBUG] [app] Starting request: urn:request:f2ee9934-df9a-41d0-9ee8-c645dd1f48ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:52,636 [241] [DEBUG] [app] Starting request: urn:request:9ffba2b0-cd2f-40d7-93a2-7f453eacb857 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:52,636 [244] [DEBUG] [app] Ending request: urn:request:f2ee9934-df9a-41d0-9ee8-c645dd1f48ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2ee9934-df9a-41d0-9ee8-c645dd1f48ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:54:52,636 [241] [DEBUG] [app] Ending request: urn:request:9ffba2b0-cd2f-40d7-93a2-7f453eacb857 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ffba2b0-cd2f-40d7-93a2-7f453eacb857', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:52,647 [243] [DEBUG] [app] Ending request: urn:request:42aac001-dedc-48bb-873e-cb602df35988 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42aac001-dedc-48bb-873e-cb602df35988', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:52,647 [239] [DEBUG] [app] Ending request: urn:request:0662155f-8a2e-49b5-a9b0-b511f6de9110 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0662155f-8a2e-49b5-a9b0-b511f6de9110', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:54:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 05:54:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:09.230038+00:00 (in 16.392226 seconds) securityworker stdout | 2023-01-17 05:54:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:22 UTC)" (scheduled at 2023-01-17 05:54:52.837365+00:00) securityworker stdout | 2023-01-17 05:54:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:22 UTC)" executed successfully builder stdout | 2023-01-17 05:54:53,974 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:53,982 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:53,983 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:53,988 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:54:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:54:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:27.226424+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 05:54:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:27 UTC)" (scheduled at 2023-01-17 05:54:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:54:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 54, 57, 227235), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:54:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:54:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:54:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:08.735066+00:00 (in 9.999496 seconds) notificationworker stdout | 2023-01-17 05:54:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:08 UTC)" (scheduled at 2023-01-17 05:54:58.735066+00:00) notificationworker stdout | 2023-01-17 05:54:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 58, 735974), True, datetime.datetime(2023, 1, 17, 5, 54, 58, 735974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:54:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:54:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:47.932490+00:00 (in 48.594610 seconds) namespacegcworker stdout | 2023-01-17 05:54:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:59 UTC)" (scheduled at 2023-01-17 05:54:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:54:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:54:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 59, 338220), True, datetime.datetime(2023, 1, 17, 5, 54, 59, 338220), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:54:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:54:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:54:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:55:01,705 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:55:01,891 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:55:02,175 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:55:02,949 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:55:03,052 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:55:03,999 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:04,007 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:04,007 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:04,013 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:55:04,275 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:55:04,951 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:55:05,075 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:55:05,365 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:55:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:55:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:35.848675+00:00 (in 29.999552 seconds) gcworker stdout | 2023-01-17 05:55:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:35 UTC)" (scheduled at 2023-01-17 05:55:05.848675+00:00) gcworker stdout | 2023-01-17 05:55:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725305849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:55:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:55:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:55:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:55:06,259 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:55:06,262 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:55:06,264 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:55:06,358 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:55:06,383 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:55:07,267 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:55:07,625 [239] [DEBUG] [app] Starting request: urn:request:eb9dd87c-db44-4629-a77a-f35c53f0f0fa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:07,625 [243] [DEBUG] [app] Starting request: urn:request:a40185da-24da-4c23-a067-a1c5728b5901 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:07,629 [232] [DEBUG] [app] Starting request: urn:request:f2ca762c-105a-4e8a-857d-04d42ab981c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,629 [242] [DEBUG] [app] Starting request: urn:request:e0967628-3fe6-4a34-b98f-608aeebdaa34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,630 [232] [DEBUG] [app] Ending request: urn:request:f2ca762c-105a-4e8a-857d-04d42ab981c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2ca762c-105a-4e8a-857d-04d42ab981c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,630 [242] [DEBUG] [app] Ending request: urn:request:e0967628-3fe6-4a34-b98f-608aeebdaa34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e0967628-3fe6-4a34-b98f-608aeebdaa34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:55:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:55:07,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:07,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,631 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,633 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:07,635 [239] [DEBUG] [app] Starting request: urn:request:b0bd0a47-136b-4f64-b619-18e4dac227fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:07,635 [239] [DEBUG] [app] Ending request: urn:request:b0bd0a47-136b-4f64-b619-18e4dac227fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b0bd0a47-136b-4f64-b619-18e4dac227fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:07,635 [244] [DEBUG] [app] Starting request: urn:request:6aaa4ac5-8d4b-441e-ae38-e6a5024aa7ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:07,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:07,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,636 [244] [DEBUG] [app] Ending request: urn:request:6aaa4ac5-8d4b-441e-ae38-e6a5024aa7ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6aaa4ac5-8d4b-441e-ae38-e6a5024aa7ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:07,640 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:07,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:07,646 [239] [DEBUG] [app] Ending request: urn:request:eb9dd87c-db44-4629-a77a-f35c53f0f0fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb9dd87c-db44-4629-a77a-f35c53f0f0fa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:07,646 [243] [DEBUG] [app] Ending request: urn:request:a40185da-24da-4c23-a067-a1c5728b5901 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a40185da-24da-4c23-a067-a1c5728b5901', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:07,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:07,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) logrotateworker stdout | 2023-01-17 05:55:08,083 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:55:08,098 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:55:08,690 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:55:08,692 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:55:08,701 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:55:08,703 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:55:08,708 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:55:08,712 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:55:08,714 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:55:08,717 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:55:08,719 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:55:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:18.735066+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 05:55:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:18 UTC)" (scheduled at 2023-01-17 05:55:08.735066+00:00) notificationworker stdout | 2023-01-17 05:55:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 8, 735845), True, datetime.datetime(2023, 1, 17, 5, 55, 8, 735845), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:55:08,795 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:55:08,798 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:55:08,801 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:55:08,803 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:55:08,805 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} storagereplication stdout | 2023-01-17 05:55:09,226 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityworker stdout | 2023-01-17 05:55:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:22.837365+00:00 (in 13.606909 seconds) securityworker stdout | 2023-01-17 05:55:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:39 UTC)" (scheduled at 2023-01-17 05:55:09.230038+00:00) securityworker stdout | 2023-01-17 05:55:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:55:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:55:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:55:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:09,856 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:55:10,622 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:55:10,654 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:55:10,698 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:55:10,807 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:55:14,023 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:14,032 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:14,032 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:14,038 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:28.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:55:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:28 UTC)" (scheduled at 2023-01-17 05:55:18.735066+00:00) notificationworker stdout | 2023-01-17 05:55:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 18, 735932), True, datetime.datetime(2023, 1, 17, 5, 55, 18, 735932), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:55:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:55:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:25.334138+00:00 (in 5.091392 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:55:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:20 UTC)" (scheduled at 2023-01-17 05:55:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:55:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:55:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 20, 243218), True, datetime.datetime(2023, 1, 17, 5, 55, 20, 243218), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:55:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:55:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:55:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:55:22,626 [239] [DEBUG] [app] Starting request: urn:request:7a3c9e2d-1767-41c0-b06a-23e5b073553b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:22,626 [244] [DEBUG] [app] Starting request: urn:request:2f7e5b95-ebbc-44e5-892f-75a1e29646c2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:22,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:22,631 [240] [DEBUG] [app] Starting request: urn:request:9b0b224d-f0bf-40bf-bab0-c672b78b83ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:22,632 [242] [DEBUG] [app] Starting request: urn:request:ed76c226-5958-4fbf-a3e3-b6359714b870 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:22,632 [240] [DEBUG] [app] Ending request: urn:request:9b0b224d-f0bf-40bf-bab0-c672b78b83ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b0b224d-f0bf-40bf-bab0-c672b78b83ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:55:22,632 [242] [DEBUG] [app] Ending request: urn:request:ed76c226-5958-4fbf-a3e3-b6359714b870 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed76c226-5958-4fbf-a3e3-b6359714b870', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:55:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:55:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:22,636 [243] [DEBUG] [app] Starting request: urn:request:1d4185f5-dd2b-4463-8b34-6773720e5220 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:22,636 [243] [DEBUG] [app] Ending request: urn:request:1d4185f5-dd2b-4463-8b34-6773720e5220 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d4185f5-dd2b-4463-8b34-6773720e5220', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:22,637 [241] [DEBUG] [app] Starting request: urn:request:06173bfe-7c2c-45a6-b3d1-db446e157974 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:22,637 [241] [DEBUG] [app] Ending request: urn:request:06173bfe-7c2c-45a6-b3d1-db446e157974 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06173bfe-7c2c-45a6-b3d1-db446e157974', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:22,648 [239] [DEBUG] [app] Ending request: urn:request:7a3c9e2d-1767-41c0-b06a-23e5b073553b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a3c9e2d-1767-41c0-b06a-23e5b073553b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:22,648 [244] [DEBUG] [app] Ending request: urn:request:2f7e5b95-ebbc-44e5-892f-75a1e29646c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f7e5b95-ebbc-44e5-892f-75a1e29646c2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:55:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:39.230038+00:00 (in 16.392221 seconds) securityworker stdout | 2023-01-17 05:55:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:52 UTC)" (scheduled at 2023-01-17 05:55:22.837365+00:00) securityworker stdout | 2023-01-17 05:55:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:52 UTC)" executed successfully builder stdout | 2023-01-17 05:55:24,038 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:24,047 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:24,047 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:24,052 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:55:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:55:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:48.830274+00:00 (in 24.188009 seconds) exportactionlogsworker stdout | 2023-01-17 05:55:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:24 UTC)" (scheduled at 2023-01-17 05:55:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:55:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:55:24,643 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:55:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:55:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:20.242198+00:00 (in 54.907515 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:55:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:25 UTC)" (scheduled at 2023-01-17 05:55:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:55:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:55:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:25 UTC)" executed successfully servicekey stdout | 2023-01-17 05:55:27,030 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 05:55:27,030 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:50:27.029240+00:00 (in 3299.999058 seconds) servicekey stdout | 2023-01-17 05:55:27,030 [88] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 06:50:27 UTC)" (scheduled at 2023-01-17 05:55:27.029240+00:00) servicekey stdout | 2023-01-17 05:55:27,030 [88] [DEBUG] [__main__] Starting automatic refresh of service key DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk to new expiration 2023-01-17 07:55:27.030577 servicekey stdout | 2023-01-17 05:55:27,031 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk', datetime.datetime(2023, 1, 10, 5, 55, 27, 30773), None, 1, 0]) servicekey stdout | 2023-01-17 05:55:27,038 [88] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443', 'DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk', 'quay', '{"n": "5tQeMffPOXC_hvyxaQcRQ0dkcGXMzwXNoRyrcogogISxmn6hWRBF7ZZ-OMVUcR_UrZF3D69owat2_XNcA6MYvIknjruzv1F0fBWfkFDLJPUdPtwG3WIPRAzDsukW0eyTXittMoUbSi_Evau-lKOQ1BA7N_EHmXI2CvDBK79zVdKmLE0gC3LLazliiE1K2j1Lpp7GRWGwP2WJoDYMAvaM6tkc0EPX-Gx-hIrfGCAcdZGotCR4ZubY9IPsF5gaLmIj7xbXWvwRdvqU0mWABxDcJs5a6uYWzk1ElGk7ITs-6lghCF_JjAJTQwwpLo1gZe89-zmPvxgpZj9kj6Lx18XqMQ", "e": "AQAB", "kty": "RSA", "kid": "DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk"}', '{"created_by": "CLI tool"}', datetime.datetime(2023, 1, 17, 4, 49, 42, 783113), datetime.datetime(2023, 1, 17, 7, 55, 27, 30577), None, 8, 8]) servicekey stdout | 2023-01-17 05:55:27,042 [88] [DEBUG] [__main__] Finished automatic refresh of service key DMXD44R8uud6BGT2LO16L4fXkpy9kvan3_vNx2cB3lk with new expiration 2023-01-17 07:55:27.030577 servicekey stdout | 2023-01-17 05:55:27,042 [88] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:55:27,042 [88] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 06:50:27 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:55:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:55:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:57.226424+00:00 (in 29.999568 seconds) buildlogsarchiver stdout | 2023-01-17 05:55:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:57 UTC)" (scheduled at 2023-01-17 05:55:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:55:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 55, 27, 227204), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:55:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:55:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:55:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:38.735066+00:00 (in 9.999478 seconds) notificationworker stdout | 2023-01-17 05:55:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:38 UTC)" (scheduled at 2023-01-17 05:55:28.735066+00:00) notificationworker stdout | 2023-01-17 05:55:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 28, 736024), True, datetime.datetime(2023, 1, 17, 5, 55, 28, 736024), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:55:31,733 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:55:31,927 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:55:32,189 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:55:32,985 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:55:33,087 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:55:34,061 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:34,070 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:34,070 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:34,076 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:55:34,293 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:55:34,980 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:55:35,080 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:55:35,382 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:55:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:55:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:05.848675+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 05:55:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:05 UTC)" (scheduled at 2023-01-17 05:55:35.848675+00:00) gcworker stdout | 2023-01-17 05:55:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725335849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:55:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:55:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:55:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:55:36,267 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:55:36,270 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:55:36,272 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:55:36,384 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:55:36,389 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:55:37,303 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:55:37,625 [244] [DEBUG] [app] Starting request: urn:request:93f68ae3-a617-4f50-8e94-4734b0e76730 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:37,625 [239] [DEBUG] [app] Starting request: urn:request:3713c67e-851d-4465-825c-d215622ee0af (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:37,630 [240] [DEBUG] [app] Starting request: urn:request:59b5138f-e92b-4598-9b1e-9685a9082712 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:37,630 [232] [DEBUG] [app] Starting request: urn:request:0495a616-39a5-4f84-9162-a7e5f4b8c00f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:37,631 [240] [DEBUG] [app] Ending request: urn:request:59b5138f-e92b-4598-9b1e-9685a9082712 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59b5138f-e92b-4598-9b1e-9685a9082712', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:55:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:55:37,631 [232] [DEBUG] [app] Ending request: urn:request:0495a616-39a5-4f84-9162-a7e5f4b8c00f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0495a616-39a5-4f84-9162-a7e5f4b8c00f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:55:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:55:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:37,635 [244] [DEBUG] [app] Starting request: urn:request:24155c08-4234-458f-866c-44dccd7e1c49 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:37,635 [243] [DEBUG] [app] Starting request: urn:request:8dc16c6c-1438-4932-805d-444a04cd2a78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:37,636 [244] [DEBUG] [app] Ending request: urn:request:24155c08-4234-458f-866c-44dccd7e1c49 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:24155c08-4234-458f-866c-44dccd7e1c49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:37,636 [243] [DEBUG] [app] Ending request: urn:request:8dc16c6c-1438-4932-805d-444a04cd2a78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8dc16c6c-1438-4932-805d-444a04cd2a78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:55:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:37,647 [244] [DEBUG] [app] Ending request: urn:request:93f68ae3-a617-4f50-8e94-4734b0e76730 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93f68ae3-a617-4f50-8e94-4734b0e76730', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:37,647 [239] [DEBUG] [app] Ending request: urn:request:3713c67e-851d-4465-825c-d215622ee0af (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3713c67e-851d-4465-825c-d215622ee0af', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) logrotateworker stdout | 2023-01-17 05:55:38,120 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:55:38,132 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:55:38,697 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:55:38,700 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:55:38,708 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:55:38,712 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:55:38,714 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:55:38,720 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:55:38,723 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:55:38,725 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:55:38,728 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:55:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:43.935629+00:00 (in 5.200086 seconds) notificationworker stdout | 2023-01-17 05:55:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:48 UTC)" (scheduled at 2023-01-17 05:55:38.735066+00:00) notificationworker stdout | 2023-01-17 05:55:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 38, 735948), True, datetime.datetime(2023, 1, 17, 5, 55, 38, 735948), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:55:38,802 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:55:38,804 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:55:38,807 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:55:38,810 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:55:38,812 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:55:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:55:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:48.738458+00:00 (in 9.707194 seconds) repositorygcworker stdout | 2023-01-17 05:55:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:39 UTC)" (scheduled at 2023-01-17 05:55:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:55:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:55:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 39, 31704), True, datetime.datetime(2023, 1, 17, 5, 55, 39, 31704), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:55:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:55:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:55:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:55:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:52.837365+00:00 (in 13.606881 seconds) securityworker stdout | 2023-01-17 05:55:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:09 UTC)" (scheduled at 2023-01-17 05:55:39.230038+00:00) securityworker stdout | 2023-01-17 05:55:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:55:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:55:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:55:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:55:39,257 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:55:39,891 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:55:40,641 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:55:40,688 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:55:40,733 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:55:40,842 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:55:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:48.735066+00:00 (in 4.798930 seconds) notificationworker stdout | 2023-01-17 05:55:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:43 UTC)" (scheduled at 2023-01-17 05:55:43.935629+00:00) notificationworker stdout | 2023-01-17 05:55:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:55:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:43 UTC)" executed successfully builder stdout | 2023-01-17 05:55:44,086 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:44,095 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:44,095 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:44,101 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:55:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:55:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:59.337413+00:00 (in 11.404495 seconds) namespacegcworker stdout | 2023-01-17 05:55:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:47 UTC)" (scheduled at 2023-01-17 05:55:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:55:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:55:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:58.735066+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:55:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:58 UTC)" (scheduled at 2023-01-17 05:55:48.735066+00:00) notificationworker stdout | 2023-01-17 05:55:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 48, 735938), True, datetime.datetime(2023, 1, 17, 5, 55, 48, 735938), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:55:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:55:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:39.030814+00:00 (in 50.291857 seconds) repositorygcworker stdout | 2023-01-17 05:55:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:48 UTC)" (scheduled at 2023-01-17 05:55:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:55:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:55:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:55:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:55:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:24.641816+00:00 (in 35.811132 seconds) exportactionlogsworker stdout | 2023-01-17 05:55:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:48 UTC)" (scheduled at 2023-01-17 05:55:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:55:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:55:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 48, 831029), True, datetime.datetime(2023, 1, 17, 5, 55, 48, 831029), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:55:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:55:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:55:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:55:52,626 [239] [DEBUG] [app] Starting request: urn:request:3878da9b-7772-4e39-b685-7127184a24bb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:52,626 [244] [DEBUG] [app] Starting request: urn:request:62e63562-d1f7-4bac-a2cf-909b9dba7716 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:55:52,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:52,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:52,631 [240] [DEBUG] [app] Starting request: urn:request:23769707-9883-4da5-b0c7-723410f53e80 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:52,632 [237] [DEBUG] [app] Starting request: urn:request:e48acec5-ce64-4338-9fb8-36ecc1b95f3b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:52,632 [240] [DEBUG] [app] Ending request: urn:request:23769707-9883-4da5-b0c7-723410f53e80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23769707-9883-4da5-b0c7-723410f53e80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:52,632 [237] [DEBUG] [app] Ending request: urn:request:e48acec5-ce64-4338-9fb8-36ecc1b95f3b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e48acec5-ce64-4338-9fb8-36ecc1b95f3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:55:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:55:52,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:52,636 [243] [DEBUG] [app] Starting request: urn:request:51419609-3c93-4a21-a94a-0ac49aa3fb6c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:52,637 [239] [DEBUG] [app] Starting request: urn:request:a0db5273-9b65-4105-b3aa-385a24d981ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:52,637 [243] [DEBUG] [app] Ending request: urn:request:51419609-3c93-4a21-a94a-0ac49aa3fb6c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:51419609-3c93-4a21-a94a-0ac49aa3fb6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:52,637 [239] [DEBUG] [app] Ending request: urn:request:a0db5273-9b65-4105-b3aa-385a24d981ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0db5273-9b65-4105-b3aa-385a24d981ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:55:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:52,649 [239] [DEBUG] [app] Ending request: urn:request:3878da9b-7772-4e39-b685-7127184a24bb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3878da9b-7772-4e39-b685-7127184a24bb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:52,649 [244] [DEBUG] [app] Ending request: urn:request:62e63562-d1f7-4bac-a2cf-909b9dba7716 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62e63562-d1f7-4bac-a2cf-909b9dba7716', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:52,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:55:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:09.230038+00:00 (in 16.392220 seconds) securityworker stdout | 2023-01-17 05:55:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:22 UTC)" (scheduled at 2023-01-17 05:55:52.837365+00:00) securityworker stdout | 2023-01-17 05:55:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:22 UTC)" executed successfully builder stdout | 2023-01-17 05:55:54,111 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:54,120 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:54,120 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:54,125 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:55:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:55:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:27.226424+00:00 (in 29.999542 seconds) buildlogsarchiver stdout | 2023-01-17 05:55:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:27 UTC)" (scheduled at 2023-01-17 05:55:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:55:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 55, 57, 227337), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:55:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:55:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:55:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:08.735066+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 05:55:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:08 UTC)" (scheduled at 2023-01-17 05:55:58.735066+00:00) notificationworker stdout | 2023-01-17 05:55:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 58, 735962), True, datetime.datetime(2023, 1, 17, 5, 55, 58, 735962), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:55:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:55:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:47.932490+00:00 (in 48.594641 seconds) namespacegcworker stdout | 2023-01-17 05:55:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:59 UTC)" (scheduled at 2023-01-17 05:55:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:55:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:55:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 59, 338258), True, datetime.datetime(2023, 1, 17, 5, 55, 59, 338258), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:55:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:55:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:55:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:56:01,744 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:56:01,961 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:56:02,215 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:56:03,020 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:56:03,093 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:56:04,136 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:04,145 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:04,145 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:04,151 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:56:04,318 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:56:05,016 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:56:05,100 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:56:05,417 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:56:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:56:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:35.848675+00:00 (in 29.999546 seconds) gcworker stdout | 2023-01-17 05:56:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:35 UTC)" (scheduled at 2023-01-17 05:56:05.848675+00:00) gcworker stdout | 2023-01-17 05:56:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725365849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:56:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:56:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:56:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:56:06,274 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:56:06,277 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:56:06,279 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:56:06,405 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:56:06,408 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:56:07,320 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:56:07,625 [239] [DEBUG] [app] Starting request: urn:request:386adf7b-4a22-4f23-b56f-b115ad9689b7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:07,625 [241] [DEBUG] [app] Starting request: urn:request:1d8dcf9f-f573-4800-99a3-2e28d31f95f5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:07,630 [240] [DEBUG] [app] Starting request: urn:request:b7b067dc-a539-4ccb-80df-f6518d603b63 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:07,631 [236] [DEBUG] [app] Starting request: urn:request:5d8d4559-1676-45be-a377-54074b2e03c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:07,631 [240] [DEBUG] [app] Ending request: urn:request:b7b067dc-a539-4ccb-80df-f6518d603b63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b7b067dc-a539-4ccb-80df-f6518d603b63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:56:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:56:07,631 [236] [DEBUG] [app] Ending request: urn:request:5d8d4559-1676-45be-a377-54074b2e03c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d8d4559-1676-45be-a377-54074b2e03c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:56:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:07,636 [244] [DEBUG] [app] Starting request: urn:request:23ed1c26-c80f-4a0f-b67a-0207eafee108 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:07,636 [241] [DEBUG] [app] Starting request: urn:request:071e647b-8204-4dd8-8407-5f2cef56374d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:07,636 [244] [DEBUG] [app] Ending request: urn:request:23ed1c26-c80f-4a0f-b67a-0207eafee108 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23ed1c26-c80f-4a0f-b67a-0207eafee108', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:07,636 [241] [DEBUG] [app] Ending request: urn:request:071e647b-8204-4dd8-8407-5f2cef56374d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:071e647b-8204-4dd8-8407-5f2cef56374d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:07,647 [241] [DEBUG] [app] Ending request: urn:request:1d8dcf9f-f573-4800-99a3-2e28d31f95f5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1d8dcf9f-f573-4800-99a3-2e28d31f95f5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:07,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:07,647 [239] [DEBUG] [app] Ending request: urn:request:386adf7b-4a22-4f23-b56f-b115ad9689b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:386adf7b-4a22-4f23-b56f-b115ad9689b7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 05:56:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) queuecleanupworker stdout | 2023-01-17 05:56:08,146 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:56:08,154 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-registry stdout | 2023-01-17 05:56:08,707 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:56:08,709 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:56:08,716 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:56:08,720 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:56:08,722 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:56:08,728 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:56:08,731 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:56:08,733 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} notificationworker stdout | 2023-01-17 05:56:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:18.735066+00:00 (in 9.999548 seconds) gunicorn-registry stdout | 2023-01-17 05:56:08,735 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} notificationworker stdout | 2023-01-17 05:56:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:18 UTC)" (scheduled at 2023-01-17 05:56:08.735066+00:00) notificationworker stdout | 2023-01-17 05:56:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 8, 735900), True, datetime.datetime(2023, 1, 17, 5, 56, 8, 735900), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:56:08,809 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:56:08,811 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:56:08,814 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:56:08,817 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:56:08,819 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:56:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:22.837365+00:00 (in 13.606889 seconds) securityworker stdout | 2023-01-17 05:56:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:39 UTC)" (scheduled at 2023-01-17 05:56:09.230038+00:00) securityworker stdout | 2023-01-17 05:56:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:56:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:56:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:56:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:56:09,263 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:56:09,927 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:56:10,662 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:56:10,723 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:56:10,768 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:56:10,848 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:56:14,160 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:14,177 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:14,177 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:14,183 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:28.735066+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 05:56:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:28 UTC)" (scheduled at 2023-01-17 05:56:18.735066+00:00) notificationworker stdout | 2023-01-17 05:56:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 18, 735978), True, datetime.datetime(2023, 1, 17, 5, 56, 18, 735978), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:56:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:56:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:25.334138+00:00 (in 5.091447 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:56:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:20 UTC)" (scheduled at 2023-01-17 05:56:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:56:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:56:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 20, 243130), True, datetime.datetime(2023, 1, 17, 5, 56, 20, 243130), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:56:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:56:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:56:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:56:22,624 [244] [DEBUG] [app] Starting request: urn:request:e4a5e59c-f47f-4e3d-b83d-9f03b8cc21ea (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:22,625 [243] [DEBUG] [app] Starting request: urn:request:109cddf1-d0e6-440e-b9e2-0c9199f57c03 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:22,629 [240] [DEBUG] [app] Starting request: urn:request:96527bc9-dc27-4bcb-8a42-bb88cd805e0b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:22,630 [232] [DEBUG] [app] Starting request: urn:request:dca656fe-2564-4526-acb3-57cde5b4bcdb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:22,630 [240] [DEBUG] [app] Ending request: urn:request:96527bc9-dc27-4bcb-8a42-bb88cd805e0b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96527bc9-dc27-4bcb-8a42-bb88cd805e0b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:22,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:56:22,630 [232] [DEBUG] [app] Ending request: urn:request:dca656fe-2564-4526-acb3-57cde5b4bcdb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dca656fe-2564-4526-acb3-57cde5b4bcdb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:22,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:56:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:22,634 [241] [DEBUG] [app] Starting request: urn:request:bdb1934a-45fb-4b4d-84c3-ab87be4b2bcd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:22,635 [241] [DEBUG] [app] Ending request: urn:request:bdb1934a-45fb-4b4d-84c3-ab87be4b2bcd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bdb1934a-45fb-4b4d-84c3-ab87be4b2bcd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:22,635 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:22,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:56:22,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:22,636 [239] [DEBUG] [app] Starting request: urn:request:7007c6e4-2369-454d-baeb-39f7bcb8571d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:22,636 [239] [DEBUG] [app] Ending request: urn:request:7007c6e4-2369-454d-baeb-39f7bcb8571d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7007c6e4-2369-454d-baeb-39f7bcb8571d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:22,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:22,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:22,646 [243] [DEBUG] [app] Ending request: urn:request:109cddf1-d0e6-440e-b9e2-0c9199f57c03 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:109cddf1-d0e6-440e-b9e2-0c9199f57c03', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:22,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:22,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:56:22,649 [244] [DEBUG] [app] Ending request: urn:request:e4a5e59c-f47f-4e3d-b83d-9f03b8cc21ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4a5e59c-f47f-4e3d-b83d-9f03b8cc21ea', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:22,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.026) securityworker stdout | 2023-01-17 05:56:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:39.230038+00:00 (in 16.392217 seconds) securityworker stdout | 2023-01-17 05:56:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:52 UTC)" (scheduled at 2023-01-17 05:56:22.837365+00:00) securityworker stdout | 2023-01-17 05:56:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:52 UTC)" executed successfully builder stdout | 2023-01-17 05:56:24,183 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:24,192 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:24,193 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:24,198 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:56:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:44.641497+00:00 (in 19.999250 seconds) exportactionlogsworker stdout | 2023-01-17 05:56:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:24 UTC)" (scheduled at 2023-01-17 05:56:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:56:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:56:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:56:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:56:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:39.333860+00:00 (in 13.999257 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:56:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:25 UTC)" (scheduled at 2023-01-17 05:56:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:56:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:56:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:56:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:56:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:57.226424+00:00 (in 29.999440 seconds) buildlogsarchiver stdout | 2023-01-17 05:56:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:57 UTC)" (scheduled at 2023-01-17 05:56:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:56:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 56, 27, 227428), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:56:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:56:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:56:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:38.735066+00:00 (in 9.999474 seconds) notificationworker stdout | 2023-01-17 05:56:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:38 UTC)" (scheduled at 2023-01-17 05:56:28.735066+00:00) notificationworker stdout | 2023-01-17 05:56:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 28, 735965), True, datetime.datetime(2023, 1, 17, 5, 56, 28, 735965), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:56:31,779 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:56:31,996 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:56:32,226 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:56:33,056 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:56:33,128 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:56:34,201 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:34,210 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:34,210 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:34,215 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:56:34,331 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 05:56:35,053 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:56:35,120 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:56:35,444 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:56:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:56:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:05.848675+00:00 (in 29.999576 seconds) gcworker stdout | 2023-01-17 05:56:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:05 UTC)" (scheduled at 2023-01-17 05:56:35.848675+00:00) gcworker stdout | 2023-01-17 05:56:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725395849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:56:35,861 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:56:35,861 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:56:35,861 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:56:36,282 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:56:36,284 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:56:36,287 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:56:36,431 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:56:36,436 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:56:37,356 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:56:37,625 [243] [DEBUG] [app] Starting request: urn:request:bd0a1f39-254a-4868-81c0-2d6ab63fdb90 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:37,626 [244] [DEBUG] [app] Starting request: urn:request:115f076e-87d7-4e34-ae4f-336050c96aee (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:37,630 [232] [DEBUG] [app] Starting request: urn:request:e4647d8f-4ea9-471b-a38c-97e8bc64eec9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,630 [232] [DEBUG] [app] Ending request: urn:request:e4647d8f-4ea9-471b-a38c-97e8bc64eec9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e4647d8f-4ea9-471b-a38c-97e8bc64eec9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,630 [242] [DEBUG] [app] Starting request: urn:request:843d91d0-f3b2-42a1-a2b5-586bb362728c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:56:37,631 [242] [DEBUG] [app] Ending request: urn:request:843d91d0-f3b2-42a1-a2b5-586bb362728c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:843d91d0-f3b2-42a1-a2b5-586bb362728c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:56:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:37,634 [244] [DEBUG] [app] Starting request: urn:request:d12ec471-edbc-4bb2-a4a8-74f3ebdf8c73 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:37,635 [244] [DEBUG] [app] Ending request: urn:request:d12ec471-edbc-4bb2-a4a8-74f3ebdf8c73 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d12ec471-edbc-4bb2-a4a8-74f3ebdf8c73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:37,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:37,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:37,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:37,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:37,637 [239] [DEBUG] [app] Starting request: urn:request:d29ddad0-dbb1-42fa-8b6e-dd299793ade5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:37,637 [239] [DEBUG] [app] Ending request: urn:request:d29ddad0-dbb1-42fa-8b6e-dd299793ade5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d29ddad0-dbb1-42fa-8b6e-dd299793ade5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:37,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:37,646 [243] [DEBUG] [app] Ending request: urn:request:bd0a1f39-254a-4868-81c0-2d6ab63fdb90 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd0a1f39-254a-4868-81c0-2d6ab63fdb90', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:37,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:37,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:56:37,647 [244] [DEBUG] [app] Ending request: urn:request:115f076e-87d7-4e34-ae4f-336050c96aee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:115f076e-87d7-4e34-ae4f-336050c96aee', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:37,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 05:56:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:56:38,161 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:56:38,170 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:56:38,717 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:56:38,720 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:56:38,725 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:56:38,727 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:56:38,729 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} notificationworker stdout | 2023-01-17 05:56:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:43.935629+00:00 (in 5.200152 seconds) notificationworker stdout | 2023-01-17 05:56:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:48 UTC)" (scheduled at 2023-01-17 05:56:38.735066+00:00) gunicorn-registry stdout | 2023-01-17 05:56:38,735 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} notificationworker stdout | 2023-01-17 05:56:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 38, 735847), True, datetime.datetime(2023, 1, 17, 5, 56, 38, 735847), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:56:38,737 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:56:38,742 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} notificationworker stdout | 2023-01-17 05:56:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:56:38,746 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:56:38,815 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:56:38,818 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:56:38,820 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:56:38,824 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:56:38,826 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:56:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:56:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:48.738458+00:00 (in 9.707180 seconds) repositorygcworker stdout | 2023-01-17 05:56:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:39 UTC)" (scheduled at 2023-01-17 05:56:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:56:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:56:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 39, 31711), True, datetime.datetime(2023, 1, 17, 5, 56, 39, 31711), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:56:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:56:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:56:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:56:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:52.837365+00:00 (in 13.606882 seconds) securityworker stdout | 2023-01-17 05:56:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:09 UTC)" (scheduled at 2023-01-17 05:56:39.230038+00:00) securityworker stdout | 2023-01-17 05:56:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:56:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:56:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:56:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:56:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:56:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:56:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:56:39,291 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} securityscanningnotificationworker stdout | 2023-01-17 05:56:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:56:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:20.242198+00:00 (in 40.907909 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:56:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:39 UTC)" (scheduled at 2023-01-17 05:56:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:56:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 56, 39, 334697), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:56:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 39, 334697), True, datetime.datetime(2023, 1, 17, 5, 56, 39, 334697), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:56:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 39, 334697), True, datetime.datetime(2023, 1, 17, 5, 56, 39, 334697), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 56, 39, 334697), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:56:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:56:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:39 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:39,957 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:56:40,696 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:56:40,758 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:56:40,795 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:56:40,874 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:56:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:48.735066+00:00 (in 4.798929 seconds) notificationworker stdout | 2023-01-17 05:56:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:43 UTC)" (scheduled at 2023-01-17 05:56:43.935629+00:00) notificationworker stdout | 2023-01-17 05:56:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:56:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:43 UTC)" executed successfully builder stdout | 2023-01-17 05:56:44,225 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:44,234 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:44,234 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:44,240 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:56:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:48.830274+00:00 (in 4.188294 seconds) exportactionlogsworker stdout | 2023-01-17 05:56:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:44 UTC)" (scheduled at 2023-01-17 05:56:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 05:56:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 56, 44, 642334), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:56:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 44, 642334), True, datetime.datetime(2023, 1, 17, 5, 56, 44, 642334), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:56:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 44, 642334), True, datetime.datetime(2023, 1, 17, 5, 56, 44, 642334), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 56, 44, 642334), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:56:44,653 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:44 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:56:47,933 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:56:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:59.337413+00:00 (in 11.404280 seconds) namespacegcworker stdout | 2023-01-17 05:56:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:47 UTC)" (scheduled at 2023-01-17 05:56:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:56:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:56:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:58.735066+00:00 (in 9.999488 seconds) notificationworker stdout | 2023-01-17 05:56:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:58 UTC)" (scheduled at 2023-01-17 05:56:48.735066+00:00) notificationworker stdout | 2023-01-17 05:56:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 48, 735972), True, datetime.datetime(2023, 1, 17, 5, 56, 48, 735972), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:56:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:56:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:04.738062+00:00 (in 15.999150 seconds) repositorygcworker stdout | 2023-01-17 05:56:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:48 UTC)" (scheduled at 2023-01-17 05:56:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:56:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:56:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:56:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:56:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:24.641816+00:00 (in 35.811108 seconds) exportactionlogsworker stdout | 2023-01-17 05:56:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:48 UTC)" (scheduled at 2023-01-17 05:56:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:56:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:56:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 48, 831070), True, datetime.datetime(2023, 1, 17, 5, 56, 48, 831070), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:56:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:56:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:56:52,625 [239] [DEBUG] [app] Starting request: urn:request:f00f8327-acfd-43bb-867e-a1bafd43a64e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:52,625 [243] [DEBUG] [app] Starting request: urn:request:7c635761-9218-4f37-810c-9f2ff1a5d3c4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:56:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:52,630 [232] [DEBUG] [app] Starting request: urn:request:df4d7edc-466a-47a4-b482-e8d88f6121df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:52,630 [242] [DEBUG] [app] Starting request: urn:request:a9a48940-1dee-44e6-9671-fda317a75c3d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:52,630 [232] [DEBUG] [app] Ending request: urn:request:df4d7edc-466a-47a4-b482-e8d88f6121df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df4d7edc-466a-47a4-b482-e8d88f6121df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:52,631 [242] [DEBUG] [app] Ending request: urn:request:a9a48940-1dee-44e6-9671-fda317a75c3d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9a48940-1dee-44e6-9671-fda317a75c3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:56:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:52,635 [244] [DEBUG] [app] Starting request: urn:request:33e75e6a-ce18-40d5-8200-35c2c03cf80b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:52,635 [239] [DEBUG] [app] Starting request: urn:request:6dddd720-3694-4f42-af4e-b14a0ae5952a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:52,636 [244] [DEBUG] [app] Ending request: urn:request:33e75e6a-ce18-40d5-8200-35c2c03cf80b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33e75e6a-ce18-40d5-8200-35c2c03cf80b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:52,636 [239] [DEBUG] [app] Ending request: urn:request:6dddd720-3694-4f42-af4e-b14a0ae5952a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6dddd720-3694-4f42-af4e-b14a0ae5952a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:52,647 [239] [DEBUG] [app] Ending request: urn:request:f00f8327-acfd-43bb-867e-a1bafd43a64e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f00f8327-acfd-43bb-867e-a1bafd43a64e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:52,647 [243] [DEBUG] [app] Ending request: urn:request:7c635761-9218-4f37-810c-9f2ff1a5d3c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c635761-9218-4f37-810c-9f2ff1a5d3c4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:56:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) securityworker stdout | 2023-01-17 05:56:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:09.230038+00:00 (in 16.392221 seconds) securityworker stdout | 2023-01-17 05:56:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:22 UTC)" (scheduled at 2023-01-17 05:56:52.837365+00:00) securityworker stdout | 2023-01-17 05:56:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:22 UTC)" executed successfully builder stdout | 2023-01-17 05:56:54,250 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:54,259 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:54,259 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:54,265 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:56:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:56:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:27.226424+00:00 (in 29.999469 seconds) buildlogsarchiver stdout | 2023-01-17 05:56:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:27 UTC)" (scheduled at 2023-01-17 05:56:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:56:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 56, 57, 227385), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:56:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:56:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:56:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:08.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 05:56:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:08 UTC)" (scheduled at 2023-01-17 05:56:58.735066+00:00) notificationworker stdout | 2023-01-17 05:56:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 58, 735964), True, datetime.datetime(2023, 1, 17, 5, 56, 58, 735964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:56:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:56:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:47.932490+00:00 (in 48.594581 seconds) namespacegcworker stdout | 2023-01-17 05:56:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:59 UTC)" (scheduled at 2023-01-17 05:56:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:56:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:56:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 59, 338305), True, datetime.datetime(2023, 1, 17, 5, 56, 59, 338305), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:56:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:56:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:56:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:57:01,786 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:57:02,031 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:57:02,234 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:57:03,092 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:57:03,135 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:57:04,275 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:04,289 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:04,290 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:04,296 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:57:04,350 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} repositorygcworker stdout | 2023-01-17 05:57:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:57:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:39.030814+00:00 (in 34.292309 seconds) repositorygcworker stdout | 2023-01-17 05:57:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:02:04 UTC)" (scheduled at 2023-01-17 05:57:04.738062+00:00) repositorygcworker stdout | 2023-01-17 05:57:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 57, 4, 738897), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:57:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 57, 4, 738897), True, datetime.datetime(2023, 1, 17, 5, 57, 4, 738897), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:57:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 57, 4, 738897), True, datetime.datetime(2023, 1, 17, 5, 57, 4, 738897), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 57, 4, 738897), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:57:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:57:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:02:04 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:57:05,069 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:57:05,154 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:57:05,480 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:57:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:57:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:35.848675+00:00 (in 29.999537 seconds) gcworker stdout | 2023-01-17 05:57:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:35 UTC)" (scheduled at 2023-01-17 05:57:05.848675+00:00) gcworker stdout | 2023-01-17 05:57:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725425849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:57:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:57:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:57:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:57:06,290 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:57:06,292 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:57:06,295 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:57:06,461 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:57:06,471 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:57:07,366 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:57:07,626 [241] [DEBUG] [app] Starting request: urn:request:812101e0-ec2d-4e9b-b2ca-9eee63a2bc27 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:07,627 [239] [DEBUG] [app] Starting request: urn:request:06bd88d1-9358-4ffb-81e1-3c37d57d76d3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:07,628 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:07,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:07,631 [242] [DEBUG] [app] Starting request: urn:request:fef25056-8631-41ff-8176-b4fdf27c8b8c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:07,632 [232] [DEBUG] [app] Starting request: urn:request:a4b02d0a-e705-4e72-a83f-d9f749701203 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:07,632 [242] [DEBUG] [app] Ending request: urn:request:fef25056-8631-41ff-8176-b4fdf27c8b8c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fef25056-8631-41ff-8176-b4fdf27c8b8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:57:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:57:07,632 [232] [DEBUG] [app] Ending request: urn:request:a4b02d0a-e705-4e72-a83f-d9f749701203 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4b02d0a-e705-4e72-a83f-d9f749701203', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:57:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:57:07,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:07,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:07,636 [243] [DEBUG] [app] Starting request: urn:request:285d3244-e56c-463a-af0b-766d77b39bd2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:07,637 [243] [DEBUG] [app] Ending request: urn:request:285d3244-e56c-463a-af0b-766d77b39bd2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:285d3244-e56c-463a-af0b-766d77b39bd2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:07,637 [241] [DEBUG] [app] Starting request: urn:request:ab5408df-ee81-4e71-b69a-919871997fa0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:07,637 [241] [DEBUG] [app] Ending request: urn:request:ab5408df-ee81-4e71-b69a-919871997fa0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ab5408df-ee81-4e71-b69a-919871997fa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:57:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:07,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:07,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:07,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:07,648 [241] [DEBUG] [app] Ending request: urn:request:812101e0-ec2d-4e9b-b2ca-9eee63a2bc27 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:812101e0-ec2d-4e9b-b2ca-9eee63a2bc27', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:07,649 [239] [DEBUG] [app] Ending request: urn:request:06bd88d1-9358-4ffb-81e1-3c37d57d76d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06bd88d1-9358-4ffb-81e1-3c37d57d76d3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:07,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:07,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:57:07,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) logrotateworker stdout | 2023-01-17 05:57:08,174 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:57:08,205 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:57:08,724 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:57:08,726 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:57:08,732 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:57:08,734 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} notificationworker stdout | 2023-01-17 05:57:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:18.735066+00:00 (in 9.999594 seconds) notificationworker stdout | 2023-01-17 05:57:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:18 UTC)" (scheduled at 2023-01-17 05:57:08.735066+00:00) notificationworker stdout | 2023-01-17 05:57:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 8, 735831), True, datetime.datetime(2023, 1, 17, 5, 57, 8, 735831), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:57:08,736 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:57:08,741 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} notificationworker stdout | 2023-01-17 05:57:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:57:08,744 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:57:08,751 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:57:08,755 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:57:08,821 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:57:08,824 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:57:08,826 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:57:08,830 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:57:08,834 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:57:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:22.837365+00:00 (in 13.606887 seconds) securityworker stdout | 2023-01-17 05:57:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:39 UTC)" (scheduled at 2023-01-17 05:57:09.230038+00:00) securityworker stdout | 2023-01-17 05:57:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:57:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:57:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:57:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:57:09,327 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:57:09,983 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:57:10,732 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:57:10,764 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:57:10,830 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:57:10,909 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:57:14,303 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:14,313 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:14,313 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:14,318 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:28.735066+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 05:57:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:28 UTC)" (scheduled at 2023-01-17 05:57:18.735066+00:00) notificationworker stdout | 2023-01-17 05:57:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 18, 735889), True, datetime.datetime(2023, 1, 17, 5, 57, 18, 735889), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:57:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:57:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:25.334138+00:00 (in 5.091376 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:57:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:20 UTC)" (scheduled at 2023-01-17 05:57:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:57:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:57:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 20, 243130), True, datetime.datetime(2023, 1, 17, 5, 57, 20, 243130), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:57:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:57:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:57:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:57:22,626 [239] [DEBUG] [app] Starting request: urn:request:132381a9-aeb9-42e6-98df-05ac027b2164 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:22,626 [243] [DEBUG] [app] Starting request: urn:request:6ea30c4d-96bc-40ba-ac98-f06eb1672625 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:22,631 [232] [DEBUG] [app] Starting request: urn:request:c6f3710c-f9dc-41c6-b283-bd4623a65bff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:22,631 [242] [DEBUG] [app] Starting request: urn:request:cccb811e-f269-4efa-9d3d-3851c1d75b51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:22,631 [232] [DEBUG] [app] Ending request: urn:request:c6f3710c-f9dc-41c6-b283-bd4623a65bff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c6f3710c-f9dc-41c6-b283-bd4623a65bff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:57:22,631 [242] [DEBUG] [app] Ending request: urn:request:cccb811e-f269-4efa-9d3d-3851c1d75b51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cccb811e-f269-4efa-9d3d-3851c1d75b51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:57:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:57:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:57:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:22,636 [241] [DEBUG] [app] Starting request: urn:request:e663b7d7-9291-4a62-b6bf-c6fb98c2e542 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:22,636 [243] [DEBUG] [app] Starting request: urn:request:480fd404-6b74-4f0d-99eb-0ad28508bcf2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:22,636 [241] [DEBUG] [app] Ending request: urn:request:e663b7d7-9291-4a62-b6bf-c6fb98c2e542 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e663b7d7-9291-4a62-b6bf-c6fb98c2e542', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:22,636 [243] [DEBUG] [app] Ending request: urn:request:480fd404-6b74-4f0d-99eb-0ad28508bcf2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:480fd404-6b74-4f0d-99eb-0ad28508bcf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:22,647 [243] [DEBUG] [app] Ending request: urn:request:6ea30c4d-96bc-40ba-ac98-f06eb1672625 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ea30c4d-96bc-40ba-ac98-f06eb1672625', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:22,648 [239] [DEBUG] [app] Ending request: urn:request:132381a9-aeb9-42e6-98df-05ac027b2164 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:132381a9-aeb9-42e6-98df-05ac027b2164', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 05:57:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 05:57:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:39.230038+00:00 (in 16.392244 seconds) securityworker stdout | 2023-01-17 05:57:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:52 UTC)" (scheduled at 2023-01-17 05:57:22.837365+00:00) securityworker stdout | 2023-01-17 05:57:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:52 UTC)" executed successfully builder stdout | 2023-01-17 05:57:24,329 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:24,337 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:24,337 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:24,343 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:57:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:57:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:48.830274+00:00 (in 24.188046 seconds) exportactionlogsworker stdout | 2023-01-17 05:57:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:24 UTC)" (scheduled at 2023-01-17 05:57:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:57:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:57:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:57:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:57:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:20.242198+00:00 (in 54.907598 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:57:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:25 UTC)" (scheduled at 2023-01-17 05:57:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:57:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:57:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:57:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:57:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:57.226424+00:00 (in 29.999495 seconds) buildlogsarchiver stdout | 2023-01-17 05:57:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:57 UTC)" (scheduled at 2023-01-17 05:57:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:57:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 57, 27, 227373), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:57:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:57:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:57:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:28,736 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:28,736 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:38.735066+00:00 (in 9.998828 seconds) notificationworker stdout | 2023-01-17 05:57:28,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:38 UTC)" (scheduled at 2023-01-17 05:57:28.735066+00:00) notificationworker stdout | 2023-01-17 05:57:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 28, 736717), True, datetime.datetime(2023, 1, 17, 5, 57, 28, 736717), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:57:31,822 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:57:32,048 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:57:32,268 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:57:33,127 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:57:33,143 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:57:34,353 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:34,362 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:34,362 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics repositorygcworker stdout | 2023-01-17 05:57:34,367 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:57:34,368 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 05:57:35,105 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:57:35,167 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:57:35,509 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:57:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:57:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:05.848675+00:00 (in 29.999433 seconds) gcworker stdout | 2023-01-17 05:57:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:05 UTC)" (scheduled at 2023-01-17 05:57:35.848675+00:00) gcworker stdout | 2023-01-17 05:57:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725455849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:57:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:57:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:57:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:57:36,297 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:57:36,300 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:57:36,302 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:57:36,497 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:57:36,499 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:57:37,381 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:57:37,626 [241] [DEBUG] [app] Starting request: urn:request:3f3f4f35-6e90-4e94-9535-43223e7d52c9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:37,626 [239] [DEBUG] [app] Starting request: urn:request:f65528ec-94d4-4033-8612-95502e8a3041 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:37,631 [240] [DEBUG] [app] Starting request: urn:request:3a2005f3-3f91-45c1-835f-6f2dcfeff2af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:37,631 [236] [DEBUG] [app] Starting request: urn:request:4181ba3c-e518-442f-b5c3-641d31b5ce51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:37,631 [240] [DEBUG] [app] Ending request: urn:request:3a2005f3-3f91-45c1-835f-6f2dcfeff2af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a2005f3-3f91-45c1-835f-6f2dcfeff2af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:37,631 [236] [DEBUG] [app] Ending request: urn:request:4181ba3c-e518-442f-b5c3-641d31b5ce51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4181ba3c-e518-442f-b5c3-641d31b5ce51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:37,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:57:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:57:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:37,636 [244] [DEBUG] [app] Starting request: urn:request:b5ea2e86-2e08-4496-b2b4-ba040223c985 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:37,636 [243] [DEBUG] [app] Starting request: urn:request:53878c0b-35f7-4700-ab7a-31c9f079da6f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:37,637 [244] [DEBUG] [app] Ending request: urn:request:b5ea2e86-2e08-4496-b2b4-ba040223c985 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b5ea2e86-2e08-4496-b2b4-ba040223c985', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:37,637 [243] [DEBUG] [app] Ending request: urn:request:53878c0b-35f7-4700-ab7a-31c9f079da6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53878c0b-35f7-4700-ab7a-31c9f079da6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:57:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:37,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:57:37,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:37,648 [239] [DEBUG] [app] Ending request: urn:request:f65528ec-94d4-4033-8612-95502e8a3041 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f65528ec-94d4-4033-8612-95502e8a3041', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:37,648 [241] [DEBUG] [app] Ending request: urn:request:3f3f4f35-6e90-4e94-9535-43223e7d52c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f3f4f35-6e90-4e94-9535-43223e7d52c9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:57:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) logrotateworker stdout | 2023-01-17 05:57:38,206 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:57:38,237 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 05:57:38,732 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:57:38,734 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} notificationworker stdout | 2023-01-17 05:57:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:43.935629+00:00 (in 5.200060 seconds) notificationworker stdout | 2023-01-17 05:57:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:48 UTC)" (scheduled at 2023-01-17 05:57:38.735066+00:00) notificationworker stdout | 2023-01-17 05:57:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 38, 735962), True, datetime.datetime(2023, 1, 17, 5, 57, 38, 735962), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:57:38,738 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:57:38,744 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} notificationworker stdout | 2023-01-17 05:57:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:57:38,746 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:57:38,751 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:57:38,754 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:57:38,759 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:57:38,762 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:57:38,828 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:57:38,831 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:57:38,834 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:57:38,836 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:57:38,839 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:57:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:57:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:48.738458+00:00 (in 9.707192 seconds) repositorygcworker stdout | 2023-01-17 05:57:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:39 UTC)" (scheduled at 2023-01-17 05:57:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:57:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:57:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 39, 31637), True, datetime.datetime(2023, 1, 17, 5, 57, 39, 31637), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:57:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:57:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:57:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:57:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:52.837365+00:00 (in 13.606914 seconds) securityworker stdout | 2023-01-17 05:57:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:09 UTC)" (scheduled at 2023-01-17 05:57:39.230038+00:00) securityworker stdout | 2023-01-17 05:57:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:57:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:57:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:57:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:57:39,362 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:57:39,989 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:57:40,758 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:57:40,798 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:57:40,844 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:57:40,943 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:57:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:48.735066+00:00 (in 4.798940 seconds) notificationworker stdout | 2023-01-17 05:57:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:43 UTC)" (scheduled at 2023-01-17 05:57:43.935629+00:00) notificationworker stdout | 2023-01-17 05:57:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:57:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:43 UTC)" executed successfully builder stdout | 2023-01-17 05:57:44,378 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:44,387 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:44,387 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:44,393 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:57:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:57:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:59.337413+00:00 (in 11.404489 seconds) namespacegcworker stdout | 2023-01-17 05:57:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:47 UTC)" (scheduled at 2023-01-17 05:57:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:57:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:57:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:58.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 05:57:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:58 UTC)" (scheduled at 2023-01-17 05:57:48.735066+00:00) notificationworker stdout | 2023-01-17 05:57:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 48, 735920), True, datetime.datetime(2023, 1, 17, 5, 57, 48, 735920), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:57:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:57:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:39.030814+00:00 (in 50.291942 seconds) repositorygcworker stdout | 2023-01-17 05:57:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:48 UTC)" (scheduled at 2023-01-17 05:57:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:57:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:57:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:57:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:57:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:24.641816+00:00 (in 35.811076 seconds) exportactionlogsworker stdout | 2023-01-17 05:57:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:48 UTC)" (scheduled at 2023-01-17 05:57:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:57:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:57:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 48, 831114), True, datetime.datetime(2023, 1, 17, 5, 57, 48, 831114), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:57:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:57:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:57:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:57:52,626 [244] [DEBUG] [app] Starting request: urn:request:340a466d-941c-466a-a6ff-6852b0184a1a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:52,626 [239] [DEBUG] [app] Starting request: urn:request:8751b4fe-edc5-4a22-a2d0-752cacf63aac (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:57:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:52,631 [232] [DEBUG] [app] Starting request: urn:request:26af86af-f64c-4512-b593-4b202c1b0241 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,631 [240] [DEBUG] [app] Starting request: urn:request:f0ede24c-f342-45a2-a78e-69909726a947 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,631 [232] [DEBUG] [app] Ending request: urn:request:26af86af-f64c-4512-b593-4b202c1b0241 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26af86af-f64c-4512-b593-4b202c1b0241', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:57:52,631 [240] [DEBUG] [app] Ending request: urn:request:f0ede24c-f342-45a2-a78e-69909726a947 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0ede24c-f342-45a2-a78e-69909726a947', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:57:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:57:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:52,636 [244] [DEBUG] [app] Starting request: urn:request:2272e88e-e0da-4cba-9a71-9ddca7e404d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:52,636 [244] [DEBUG] [app] Ending request: urn:request:2272e88e-e0da-4cba-9a71-9ddca7e404d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2272e88e-e0da-4cba-9a71-9ddca7e404d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:52,638 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:52,638 [243] [DEBUG] [app] Starting request: urn:request:eb9b0ef0-7105-46a1-8f76-bd0ba652a366 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:52,639 [243] [DEBUG] [app] Ending request: urn:request:eb9b0ef0-7105-46a1-8f76-bd0ba652a366 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb9b0ef0-7105-46a1-8f76-bd0ba652a366', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:52,639 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:57:52,639 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:52,640 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:52,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:52,648 [239] [DEBUG] [app] Ending request: urn:request:8751b4fe-edc5-4a22-a2d0-752cacf63aac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8751b4fe-edc5-4a22-a2d0-752cacf63aac', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:57:52,650 [244] [DEBUG] [app] Ending request: urn:request:340a466d-941c-466a-a6ff-6852b0184a1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:340a466d-941c-466a-a6ff-6852b0184a1a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:52,650 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:52,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) securityworker stdout | 2023-01-17 05:57:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:09.230038+00:00 (in 16.392254 seconds) securityworker stdout | 2023-01-17 05:57:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:22 UTC)" (scheduled at 2023-01-17 05:57:52.837365+00:00) securityworker stdout | 2023-01-17 05:57:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:22 UTC)" executed successfully builder stdout | 2023-01-17 05:57:54,403 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:54,413 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:54,413 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:54,418 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:57:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:57:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:27.226424+00:00 (in 29.999495 seconds) buildlogsarchiver stdout | 2023-01-17 05:57:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:27 UTC)" (scheduled at 2023-01-17 05:57:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:57:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 57, 57, 227398), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:57:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:57:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:57:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:08.735066+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 05:57:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:08 UTC)" (scheduled at 2023-01-17 05:57:58.735066+00:00) notificationworker stdout | 2023-01-17 05:57:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 58, 735969), True, datetime.datetime(2023, 1, 17, 5, 57, 58, 735969), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:57:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:57:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:47.932490+00:00 (in 48.594634 seconds) namespacegcworker stdout | 2023-01-17 05:57:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:59 UTC)" (scheduled at 2023-01-17 05:57:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:57:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:57:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 59, 338254), True, datetime.datetime(2023, 1, 17, 5, 57, 59, 338254), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:57:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:57:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:57:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:58:01,845 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:58:02,080 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:58:02,277 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 05:58:03,149 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 05:58:03,156 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 05:58:04,380 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:58:04,428 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:04,438 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:04,438 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:04,444 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 05:58:05,140 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:58:05,189 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:58:05,517 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:58:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:58:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:35.848675+00:00 (in 29.999533 seconds) gcworker stdout | 2023-01-17 05:58:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:35 UTC)" (scheduled at 2023-01-17 05:58:05.848675+00:00) gcworker stdout | 2023-01-17 05:58:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725485849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:58:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:58:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:58:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:58:06,306 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:58:06,308 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:58:06,310 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} globalpromstats stdout | 2023-01-17 05:58:06,533 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 05:58:06,535 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 05:58:07,413 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:58:07,626 [239] [DEBUG] [app] Starting request: urn:request:cedf4eb0-05d4-496c-9fe6-6476dfd88361 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:07,626 [244] [DEBUG] [app] Starting request: urn:request:cab69809-849b-452a-8a69-2e0f4ed97f92 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:07,630 [236] [DEBUG] [app] Starting request: urn:request:6ba36161-b269-4a20-9a53-3e1d82b37be6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:07,631 [236] [DEBUG] [app] Ending request: urn:request:6ba36161-b269-4a20-9a53-3e1d82b37be6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ba36161-b269-4a20-9a53-3e1d82b37be6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:07,631 [235] [DEBUG] [app] Starting request: urn:request:d96bfa7e-314a-4ab3-a367-70b13886fae7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:07,631 [235] [DEBUG] [app] Ending request: urn:request:d96bfa7e-314a-4ab3-a367-70b13886fae7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d96bfa7e-314a-4ab3-a367-70b13886fae7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:58:07,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:07,636 [239] [DEBUG] [app] Starting request: urn:request:f9e35b44-678e-421a-abd0-02a79a756539 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:07,636 [239] [DEBUG] [app] Ending request: urn:request:f9e35b44-678e-421a-abd0-02a79a756539 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9e35b44-678e-421a-abd0-02a79a756539', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:07,637 [243] [DEBUG] [app] Starting request: urn:request:a4277bc1-54d4-47cc-9a2e-cf874e44b436 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:07,637 [243] [DEBUG] [app] Ending request: urn:request:a4277bc1-54d4-47cc-9a2e-cf874e44b436 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4277bc1-54d4-47cc-9a2e-cf874e44b436', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:07,647 [239] [DEBUG] [app] Ending request: urn:request:cedf4eb0-05d4-496c-9fe6-6476dfd88361 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cedf4eb0-05d4-496c-9fe6-6476dfd88361', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:07,648 [244] [DEBUG] [app] Ending request: urn:request:cab69809-849b-452a-8a69-2e0f4ed97f92 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cab69809-849b-452a-8a69-2e0f4ed97f92', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 05:58:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" logrotateworker stdout | 2023-01-17 05:58:08,228 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:58:08,272 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 05:58:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:18.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:58:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:18 UTC)" (scheduled at 2023-01-17 05:58:08.735066+00:00) notificationworker stdout | 2023-01-17 05:58:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 8, 735910), True, datetime.datetime(2023, 1, 17, 5, 58, 8, 735910), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:58:08,739 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:58:08,743 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:58:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:58:08,746 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:58:08,753 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:58:08,756 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:58:08,761 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:58:08,763 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:58:08,768 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:58:08,770 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:58:08,835 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:58:08,838 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:58:08,840 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:58:08,842 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:58:08,845 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:58:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:22.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 05:58:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:39 UTC)" (scheduled at 2023-01-17 05:58:09.230038+00:00) securityworker stdout | 2023-01-17 05:58:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:58:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:58:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:58:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:58:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:58:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:58:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:58:09,397 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:58:09,998 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:58:10,782 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:58:10,830 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:58:10,878 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:58:10,965 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:58:14,454 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:14,463 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:14,463 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:14,469 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:28.735066+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 05:58:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:28 UTC)" (scheduled at 2023-01-17 05:58:18.735066+00:00) notificationworker stdout | 2023-01-17 05:58:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 18, 735916), True, datetime.datetime(2023, 1, 17, 5, 58, 18, 735916), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:58:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:58:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:25.334138+00:00 (in 5.091479 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:58:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:20 UTC)" (scheduled at 2023-01-17 05:58:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:58:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:58:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 20, 243083), True, datetime.datetime(2023, 1, 17, 5, 58, 20, 243083), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:58:20,261 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:58:20,261 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:58:20,261 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:58:22,625 [239] [DEBUG] [app] Starting request: urn:request:75fab721-c869-458f-aa89-4aa516da9d72 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:22,625 [244] [DEBUG] [app] Starting request: urn:request:1038be98-0af8-4674-9f3e-2f463e1c82be (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:22,630 [232] [DEBUG] [app] Starting request: urn:request:929147d5-e876-407a-b84b-c4946cc28b37 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,630 [232] [DEBUG] [app] Ending request: urn:request:929147d5-e876-407a-b84b-c4946cc28b37 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:929147d5-e876-407a-b84b-c4946cc28b37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,630 [240] [DEBUG] [app] Starting request: urn:request:14c44aea-d606-4015-ad5e-2098c505d407 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:58:22,631 [240] [DEBUG] [app] Ending request: urn:request:14c44aea-d606-4015-ad5e-2098c505d407 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:14c44aea-d606-4015-ad5e-2098c505d407', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:58:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:22,635 [244] [DEBUG] [app] Starting request: urn:request:b19fb09f-ad16-432b-b783-0416bcb5c7d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:22,635 [244] [DEBUG] [app] Ending request: urn:request:b19fb09f-ad16-432b-b783-0416bcb5c7d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b19fb09f-ad16-432b-b783-0416bcb5c7d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:22,637 [243] [DEBUG] [app] Starting request: urn:request:27981911-f1db-43cd-9411-a2a5348f1bac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:22,638 [243] [DEBUG] [app] Ending request: urn:request:27981911-f1db-43cd-9411-a2a5348f1bac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27981911-f1db-43cd-9411-a2a5348f1bac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:58:22,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:22,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:22,647 [239] [DEBUG] [app] Ending request: urn:request:75fab721-c869-458f-aa89-4aa516da9d72 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75fab721-c869-458f-aa89-4aa516da9d72', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 05:58:22,649 [244] [DEBUG] [app] Ending request: urn:request:1038be98-0af8-4674-9f3e-2f463e1c82be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1038be98-0af8-4674-9f3e-2f463e1c82be', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) securityworker stdout | 2023-01-17 05:58:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:39.230038+00:00 (in 16.392231 seconds) securityworker stdout | 2023-01-17 05:58:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:52 UTC)" (scheduled at 2023-01-17 05:58:22.837365+00:00) securityworker stdout | 2023-01-17 05:58:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:52 UTC)" executed successfully builder stdout | 2023-01-17 05:58:24,477 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:24,486 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:24,486 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:24,492 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:58:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:58:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:48.830274+00:00 (in 24.188058 seconds) exportactionlogsworker stdout | 2023-01-17 05:58:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:24 UTC)" (scheduled at 2023-01-17 05:58:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:58:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:58:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:58:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:58:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:20.242198+00:00 (in 54.907578 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:58:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:25 UTC)" (scheduled at 2023-01-17 05:58:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:58:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:58:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:58:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:58:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:57.226424+00:00 (in 29.999462 seconds) buildlogsarchiver stdout | 2023-01-17 05:58:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:57 UTC)" (scheduled at 2023-01-17 05:58:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:58:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 58, 27, 227386), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:58:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:58:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:58:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:38.735066+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:58:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:38 UTC)" (scheduled at 2023-01-17 05:58:28.735066+00:00) notificationworker stdout | 2023-01-17 05:58:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 28, 735912), True, datetime.datetime(2023, 1, 17, 5, 58, 28, 735912), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:58:31,880 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:58:32,115 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:58:32,312 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:58:33,168 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:58:33,184 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:58:34,414 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:58:34,493 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:34,502 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:34,502 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:34,507 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 05:58:35,176 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:58:35,194 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:58:35,552 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:58:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:58:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:05.848675+00:00 (in 29.999518 seconds) gcworker stdout | 2023-01-17 05:58:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:05 UTC)" (scheduled at 2023-01-17 05:58:35.848675+00:00) gcworker stdout | 2023-01-17 05:58:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725515849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:58:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:58:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:58:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:58:36,313 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:58:36,315 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:58:36,318 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:58:36,565 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:58:36,567 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:58:37,448 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:58:37,625 [244] [DEBUG] [app] Starting request: urn:request:4bb3d572-6102-4640-b74a-bb729912caa6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:37,625 [239] [DEBUG] [app] Starting request: urn:request:594c3561-a86c-441b-8263-354e048c8e41 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:37,630 [240] [DEBUG] [app] Starting request: urn:request:771e8f50-b773-492c-9e50-9b9b4e447517 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:37,630 [240] [DEBUG] [app] Ending request: urn:request:771e8f50-b773-492c-9e50-9b9b4e447517 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:771e8f50-b773-492c-9e50-9b9b4e447517', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:37,630 [237] [DEBUG] [app] Starting request: urn:request:1805bf90-0539-445f-a918-53e88901938d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:37,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:58:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:58:37,631 [237] [DEBUG] [app] Ending request: urn:request:1805bf90-0539-445f-a918-53e88901938d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1805bf90-0539-445f-a918-53e88901938d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:37,635 [243] [DEBUG] [app] Starting request: urn:request:4b85a957-483c-444a-8cfe-44b351d2709a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:37,636 [243] [DEBUG] [app] Ending request: urn:request:4b85a957-483c-444a-8cfe-44b351d2709a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b85a957-483c-444a-8cfe-44b351d2709a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:37,636 [239] [DEBUG] [app] Starting request: urn:request:202e6125-a396-4be6-bde3-fd58444eaccd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:37,636 [239] [DEBUG] [app] Ending request: urn:request:202e6125-a396-4be6-bde3-fd58444eaccd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:202e6125-a396-4be6-bde3-fd58444eaccd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:37,647 [244] [DEBUG] [app] Ending request: urn:request:4bb3d572-6102-4640-b74a-bb729912caa6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4bb3d572-6102-4640-b74a-bb729912caa6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:37,647 [239] [DEBUG] [app] Ending request: urn:request:594c3561-a86c-441b-8263-354e048c8e41 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:594c3561-a86c-441b-8263-354e048c8e41', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:58:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) logrotateworker stdout | 2023-01-17 05:58:38,243 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:58:38,277 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 05:58:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:43.935629+00:00 (in 5.200126 seconds) notificationworker stdout | 2023-01-17 05:58:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:48 UTC)" (scheduled at 2023-01-17 05:58:38.735066+00:00) notificationworker stdout | 2023-01-17 05:58:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 38, 735871), True, datetime.datetime(2023, 1, 17, 5, 58, 38, 735871), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:58:38,751 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:58:38,754 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:58:38,756 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:58:38,762 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:58:38,765 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:58:38,770 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:58:38,772 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:58:38,774 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:58:38,778 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:58:38,842 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:58:38,845 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:58:38,847 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:58:38,850 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:58:38,852 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:58:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:58:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:48.738458+00:00 (in 9.707194 seconds) repositorygcworker stdout | 2023-01-17 05:58:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:39 UTC)" (scheduled at 2023-01-17 05:58:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:58:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:58:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 39, 31701), True, datetime.datetime(2023, 1, 17, 5, 58, 39, 31701), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:58:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:58:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:58:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:58:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:52.837365+00:00 (in 13.606865 seconds) securityworker stdout | 2023-01-17 05:58:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:09 UTC)" (scheduled at 2023-01-17 05:58:39.230038+00:00) securityworker stdout | 2023-01-17 05:58:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:58:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:58:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:58:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:58:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:58:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:58:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:58:39,430 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:58:40,024 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:58:40,818 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:58:40,865 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:58:40,888 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:58:40,978 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:58:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:48.735066+00:00 (in 4.798984 seconds) notificationworker stdout | 2023-01-17 05:58:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:43 UTC)" (scheduled at 2023-01-17 05:58:43.935629+00:00) notificationworker stdout | 2023-01-17 05:58:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:58:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:43 UTC)" executed successfully builder stdout | 2023-01-17 05:58:44,517 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:44,527 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:44,527 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:44,532 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:58:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:58:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:59.337413+00:00 (in 11.404453 seconds) namespacegcworker stdout | 2023-01-17 05:58:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:47 UTC)" (scheduled at 2023-01-17 05:58:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:58:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:58:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:56.935055+00:00 (in 8.199534 seconds) notificationworker stdout | 2023-01-17 05:58:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:58 UTC)" (scheduled at 2023-01-17 05:58:48.735066+00:00) notificationworker stdout | 2023-01-17 05:58:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 48, 735937), True, datetime.datetime(2023, 1, 17, 5, 58, 48, 735937), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:58:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:58:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:39.030814+00:00 (in 50.291902 seconds) repositorygcworker stdout | 2023-01-17 05:58:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:48 UTC)" (scheduled at 2023-01-17 05:58:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:58:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:58:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:58:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:58:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:24.641816+00:00 (in 35.811090 seconds) exportactionlogsworker stdout | 2023-01-17 05:58:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:48 UTC)" (scheduled at 2023-01-17 05:58:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:58:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:58:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 48, 831122), True, datetime.datetime(2023, 1, 17, 5, 58, 48, 831122), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:58:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:58:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:58:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:58:52,624 [239] [DEBUG] [app] Starting request: urn:request:8f05b43d-03d8-49f4-8e44-61403519c415 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:52,624 [243] [DEBUG] [app] Starting request: urn:request:6482f429-db50-488f-8c96-5c9f2d467e10 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:58:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:52,629 [242] [DEBUG] [app] Starting request: urn:request:a29d6c5b-d10f-4dfa-a472-442bbce9e9e7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,630 [240] [DEBUG] [app] Starting request: urn:request:846193d5-fc19-48c9-8dd2-ecbf59438b92 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,630 [242] [DEBUG] [app] Ending request: urn:request:a29d6c5b-d10f-4dfa-a472-442bbce9e9e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a29d6c5b-d10f-4dfa-a472-442bbce9e9e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,630 [240] [DEBUG] [app] Ending request: urn:request:846193d5-fc19-48c9-8dd2-ecbf59438b92 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:846193d5-fc19-48c9-8dd2-ecbf59438b92', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:58:52,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:52,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:52,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:52,631 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,633 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:52,635 [244] [DEBUG] [app] Starting request: urn:request:e5031c78-fe4a-4ca4-8033-d2effe94ad7f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:52,635 [239] [DEBUG] [app] Starting request: urn:request:8413b138-ea7a-4c1d-aef6-a164e2c7df56 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:52,635 [244] [DEBUG] [app] Ending request: urn:request:e5031c78-fe4a-4ca4-8033-d2effe94ad7f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5031c78-fe4a-4ca4-8033-d2effe94ad7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:52,635 [239] [DEBUG] [app] Ending request: urn:request:8413b138-ea7a-4c1d-aef6-a164e2c7df56 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8413b138-ea7a-4c1d-aef6-a164e2c7df56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:52,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:52,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:52,646 [239] [DEBUG] [app] Ending request: urn:request:8f05b43d-03d8-49f4-8e44-61403519c415 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f05b43d-03d8-49f4-8e44-61403519c415', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:52,647 [243] [DEBUG] [app] Ending request: urn:request:6482f429-db50-488f-8c96-5c9f2d467e10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6482f429-db50-488f-8c96-5c9f2d467e10', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:58:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:58:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:09.230038+00:00 (in 16.392258 seconds) securityworker stdout | 2023-01-17 05:58:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:22 UTC)" (scheduled at 2023-01-17 05:58:52.837365+00:00) securityworker stdout | 2023-01-17 05:58:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:22 UTC)" executed successfully builder stdout | 2023-01-17 05:58:54,543 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:54,552 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:54,552 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:54,558 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:58.735066+00:00 (in 1.799561 seconds) notificationworker stdout | 2023-01-17 05:58:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:03:56 UTC)" (scheduled at 2023-01-17 05:58:56.935055+00:00) notificationworker stdout | 2023-01-17 05:58:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 58, 56, 935858), 'notification/%']) notificationworker stdout | 2023-01-17 05:58:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 58, 56, 935858), True, datetime.datetime(2023, 1, 17, 5, 58, 56, 935858), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:58:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 58, 56, 935858), True, datetime.datetime(2023, 1, 17, 5, 58, 56, 935858), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 58, 56, 935858), 'notification/%']) notificationworker stdout | 2023-01-17 05:58:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:03:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:58:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:58:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:27.226424+00:00 (in 29.999501 seconds) buildlogsarchiver stdout | 2023-01-17 05:58:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:27 UTC)" (scheduled at 2023-01-17 05:58:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:58:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 58, 57, 227265), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:58:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:58:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:58:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:08.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:58:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:08 UTC)" (scheduled at 2023-01-17 05:58:58.735066+00:00) notificationworker stdout | 2023-01-17 05:58:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 58, 735938), True, datetime.datetime(2023, 1, 17, 5, 58, 58, 735938), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:58,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:58,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:58:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:58:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:15.931994+00:00 (in 16.594051 seconds) namespacegcworker stdout | 2023-01-17 05:58:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:59 UTC)" (scheduled at 2023-01-17 05:58:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:58:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:58:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 59, 338368), True, datetime.datetime(2023, 1, 17, 5, 58, 59, 338368), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:58:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:58:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:58:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:59 UTC)" executed successfully securityworker stdout | 2023-01-17 05:59:01,899 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:59:02,150 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:59:02,328 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:59:03,183 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:59:03,219 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:59:04,443 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:59:04,568 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:04,577 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:04,577 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:04,583 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 05:59:05,194 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 05:59:05,215 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:59:05,588 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:59:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:59:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:35.848675+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 05:59:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:35 UTC)" (scheduled at 2023-01-17 05:59:05.848675+00:00) gcworker stdout | 2023-01-17 05:59:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725545849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:59:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:59:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:59:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:59:06,320 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:59:06,322 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:59:06,325 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:59:06,578 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:59:06,601 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:59:07,484 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:59:07,625 [243] [DEBUG] [app] Starting request: urn:request:85a2692b-4e62-4f2c-bd3b-e22099c21958 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:07,625 [244] [DEBUG] [app] Starting request: urn:request:04fa5703-1bfb-4c45-a571-002311bdfc91 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:07,630 [232] [DEBUG] [app] Starting request: urn:request:0c9f3f8e-5547-405b-8af4-2b7be468d7a9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:07,630 [242] [DEBUG] [app] Starting request: urn:request:584aef71-8c2c-4220-927c-f7b9c8c172cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:07,630 [232] [DEBUG] [app] Ending request: urn:request:0c9f3f8e-5547-405b-8af4-2b7be468d7a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c9f3f8e-5547-405b-8af4-2b7be468d7a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:59:07,631 [242] [DEBUG] [app] Ending request: urn:request:584aef71-8c2c-4220-927c-f7b9c8c172cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:584aef71-8c2c-4220-927c-f7b9c8c172cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:59:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:59:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:07,635 [241] [DEBUG] [app] Starting request: urn:request:9c5df193-d15e-4d93-9eff-4fd2e8730dc8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:07,635 [241] [DEBUG] [app] Ending request: urn:request:9c5df193-d15e-4d93-9eff-4fd2e8730dc8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9c5df193-d15e-4d93-9eff-4fd2e8730dc8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:07,635 [239] [DEBUG] [app] Starting request: urn:request:11563b91-4bfe-426b-ae1e-ced2f668110c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:07,636 [239] [DEBUG] [app] Ending request: urn:request:11563b91-4bfe-426b-ae1e-ced2f668110c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11563b91-4bfe-426b-ae1e-ced2f668110c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:59:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:07,646 [243] [DEBUG] [app] Ending request: urn:request:85a2692b-4e62-4f2c-bd3b-e22099c21958 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:85a2692b-4e62-4f2c-bd3b-e22099c21958', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:07,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:07,646 [244] [DEBUG] [app] Ending request: urn:request:04fa5703-1bfb-4c45-a571-002311bdfc91 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04fa5703-1bfb-4c45-a571-002311bdfc91', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:07,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:07,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 05:59:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) logrotateworker stdout | 2023-01-17 05:59:08,280 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:59:08,294 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 05:59:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:18.735066+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 05:59:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:18 UTC)" (scheduled at 2023-01-17 05:59:08.735066+00:00) notificationworker stdout | 2023-01-17 05:59:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 8, 735927), True, datetime.datetime(2023, 1, 17, 5, 59, 8, 735927), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:59:08,758 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:59:08,762 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:59:08,765 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:59:08,770 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:59:08,772 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:59:08,779 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:59:08,782 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:59:08,784 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:59:08,786 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:59:08,849 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:59:08,852 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:59:08,855 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:59:08,857 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:59:08,860 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:59:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:22.837365+00:00 (in 13.606838 seconds) securityworker stdout | 2023-01-17 05:59:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:39 UTC)" (scheduled at 2023-01-17 05:59:09.230038+00:00) securityworker stdout | 2023-01-17 05:59:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:59:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:59:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:59:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:59:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:59:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:59:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:59:09,444 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:59:10,060 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:59:10,843 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:59:10,900 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:59:10,922 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:59:10,999 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 05:59:14,591 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:14,600 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:14,600 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:14,606 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:59:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:59:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:47.932490+00:00 (in 31.999976 seconds) namespacegcworker stdout | 2023-01-17 05:59:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:04:15 UTC)" (scheduled at 2023-01-17 05:59:15.931994+00:00) namespacegcworker stdout | 2023-01-17 05:59:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 59, 15, 932849), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:59:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 59, 15, 932849), True, datetime.datetime(2023, 1, 17, 5, 59, 15, 932849), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:59:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 59, 15, 932849), True, datetime.datetime(2023, 1, 17, 5, 59, 15, 932849), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 59, 15, 932849), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:59:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:59:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:04:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:28.735066+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 05:59:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:28 UTC)" (scheduled at 2023-01-17 05:59:18.735066+00:00) notificationworker stdout | 2023-01-17 05:59:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 18, 735964), True, datetime.datetime(2023, 1, 17, 5, 59, 18, 735964), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:59:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:59:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:25.334138+00:00 (in 5.091491 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:59:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:20 UTC)" (scheduled at 2023-01-17 05:59:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:59:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:59:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 20, 243025), True, datetime.datetime(2023, 1, 17, 5, 59, 20, 243025), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:59:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:59:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:59:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:59:22,625 [244] [DEBUG] [app] Starting request: urn:request:ec25835d-c2a7-4424-a3f8-6f4c3c2d9fe3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:22,625 [239] [DEBUG] [app] Starting request: urn:request:014d175c-ac6c-4713-8dc8-608eade43966 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:22,629 [242] [DEBUG] [app] Starting request: urn:request:8709e79f-9454-4171-8b21-9fb4cd04fae6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:22,630 [232] [DEBUG] [app] Starting request: urn:request:914ffed0-387c-453d-897d-e6b09ac43a97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:22,630 [242] [DEBUG] [app] Ending request: urn:request:8709e79f-9454-4171-8b21-9fb4cd04fae6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8709e79f-9454-4171-8b21-9fb4cd04fae6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:59:22,630 [232] [DEBUG] [app] Ending request: urn:request:914ffed0-387c-453d-897d-e6b09ac43a97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:914ffed0-387c-453d-897d-e6b09ac43a97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:22,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:22,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:22,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:59:22,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:22,631 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,633 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:22,634 [243] [DEBUG] [app] Starting request: urn:request:8e2650f5-8f8e-4621-a9eb-391f7cde5ee1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:22,634 [241] [DEBUG] [app] Starting request: urn:request:7957fb87-59ac-4eca-a7b6-77cdfb9a32ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:22,635 [243] [DEBUG] [app] Ending request: urn:request:8e2650f5-8f8e-4621-a9eb-391f7cde5ee1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e2650f5-8f8e-4621-a9eb-391f7cde5ee1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:22,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:22,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:22,635 [241] [DEBUG] [app] Ending request: urn:request:7957fb87-59ac-4eca-a7b6-77cdfb9a32ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7957fb87-59ac-4eca-a7b6-77cdfb9a32ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:22,635 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:59:22,635 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:22,635 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:22,640 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:22,640 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:22,646 [239] [DEBUG] [app] Ending request: urn:request:014d175c-ac6c-4713-8dc8-608eade43966 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:014d175c-ac6c-4713-8dc8-608eade43966', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:22,646 [244] [DEBUG] [app] Ending request: urn:request:ec25835d-c2a7-4424-a3f8-6f4c3c2d9fe3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec25835d-c2a7-4424-a3f8-6f4c3c2d9fe3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:22,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:22,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:22,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:59:22,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 05:59:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:39.230038+00:00 (in 16.392201 seconds) securityworker stdout | 2023-01-17 05:59:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:52 UTC)" (scheduled at 2023-01-17 05:59:22.837365+00:00) securityworker stdout | 2023-01-17 05:59:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:52 UTC)" executed successfully builder stdout | 2023-01-17 05:59:24,609 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:24,618 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:24,618 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:24,624 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:59:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:59:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:48.830274+00:00 (in 24.188033 seconds) exportactionlogsworker stdout | 2023-01-17 05:59:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:24 UTC)" (scheduled at 2023-01-17 05:59:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 05:59:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:59:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:59:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:59:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:20.242198+00:00 (in 54.907616 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:59:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:25 UTC)" (scheduled at 2023-01-17 05:59:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:59:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:59:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:59:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:59:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:57.226424+00:00 (in 29.999583 seconds) buildlogsarchiver stdout | 2023-01-17 05:59:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:57 UTC)" (scheduled at 2023-01-17 05:59:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:59:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 59, 27, 227386), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:59:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:59:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:59:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:38.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 05:59:28,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:38 UTC)" (scheduled at 2023-01-17 05:59:28.735066+00:00) notificationworker stdout | 2023-01-17 05:59:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 28, 736198), True, datetime.datetime(2023, 1, 17, 5, 59, 28, 736198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:38 UTC)" executed successfully securityworker stdout | 2023-01-17 05:59:31,935 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 05:59:32,185 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 05:59:32,347 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 05:59:33,217 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 05:59:33,254 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:59:34,477 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 05:59:34,625 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:34,633 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:34,633 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:34,639 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 05:59:35,230 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 05:59:35,232 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:59:35,623 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 05:59:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:59:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:05.848675+00:00 (in 29.999563 seconds) gcworker stdout | 2023-01-17 05:59:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:05 UTC)" (scheduled at 2023-01-17 05:59:35.848675+00:00) gcworker stdout | 2023-01-17 05:59:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725575849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:59:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:59:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:59:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:59:36,328 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 05:59:36,330 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:59:36,332 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 05:59:36,584 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 05:59:36,628 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 05:59:37,494 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:59:37,625 [239] [DEBUG] [app] Starting request: urn:request:99ec4ae1-ed72-4a55-8f02-134a03be1772 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:37,626 [244] [DEBUG] [app] Starting request: urn:request:73ce7a8a-7314-42a9-a517-3695135c074d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:37,630 [232] [DEBUG] [app] Starting request: urn:request:a849ca68-4d42-488d-9ab1-66a0f32f99ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:37,631 [240] [DEBUG] [app] Starting request: urn:request:7b46cb55-0774-44c9-b79f-9030557d8a97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:37,631 [232] [DEBUG] [app] Ending request: urn:request:a849ca68-4d42-488d-9ab1-66a0f32f99ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a849ca68-4d42-488d-9ab1-66a0f32f99ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:59:37,631 [240] [DEBUG] [app] Ending request: urn:request:7b46cb55-0774-44c9-b79f-9030557d8a97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b46cb55-0774-44c9-b79f-9030557d8a97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:59:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:37,636 [243] [DEBUG] [app] Starting request: urn:request:e6627ffa-90bc-4e67-8ac1-285f1345e586 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:37,636 [243] [DEBUG] [app] Ending request: urn:request:e6627ffa-90bc-4e67-8ac1-285f1345e586 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6627ffa-90bc-4e67-8ac1-285f1345e586', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:37,636 [241] [DEBUG] [app] Starting request: urn:request:dc20f4b8-5791-4998-8b71-75a5fbb0c1e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:37,637 [241] [DEBUG] [app] Ending request: urn:request:dc20f4b8-5791-4998-8b71-75a5fbb0c1e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc20f4b8-5791-4998-8b71-75a5fbb0c1e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:59:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:37,647 [244] [DEBUG] [app] Ending request: urn:request:73ce7a8a-7314-42a9-a517-3695135c074d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:73ce7a8a-7314-42a9-a517-3695135c074d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:37,647 [239] [DEBUG] [app] Ending request: urn:request:99ec4ae1-ed72-4a55-8f02-134a03be1772 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99ec4ae1-ed72-4a55-8f02-134a03be1772', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:59:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) logrotateworker stdout | 2023-01-17 05:59:38,316 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 05:59:38,328 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 05:59:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:43.935629+00:00 (in 5.200136 seconds) notificationworker stdout | 2023-01-17 05:59:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:48 UTC)" (scheduled at 2023-01-17 05:59:38.735066+00:00) notificationworker stdout | 2023-01-17 05:59:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 38, 735880), True, datetime.datetime(2023, 1, 17, 5, 59, 38, 735880), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:59:38,768 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:59:38,771 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 05:59:38,773 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:59:38,779 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 05:59:38,782 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:59:38,787 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:59:38,789 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:59:38,791 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 05:59:38,794 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 05:59:38,855 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 05:59:38,859 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 05:59:38,863 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 05:59:38,865 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 05:59:38,867 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 05:59:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:59:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:48.738458+00:00 (in 9.707231 seconds) repositorygcworker stdout | 2023-01-17 05:59:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:39 UTC)" (scheduled at 2023-01-17 05:59:39.030814+00:00) repositorygcworker stdout | 2023-01-17 05:59:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:59:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 39, 31625), True, datetime.datetime(2023, 1, 17, 5, 59, 39, 31625), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:59:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:59:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:59:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:39 UTC)" executed successfully securityworker stdout | 2023-01-17 05:59:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:52.837365+00:00 (in 13.606902 seconds) securityworker stdout | 2023-01-17 05:59:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:09 UTC)" (scheduled at 2023-01-17 05:59:39.230038+00:00) securityworker stdout | 2023-01-17 05:59:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:59:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:59:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:39,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:39,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:59:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:59:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:59:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:59:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:59:39,480 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:59:40,080 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:59:40,852 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 05:59:40,936 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 05:59:40,939 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:59:41,034 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 05:59:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:48.735066+00:00 (in 4.798958 seconds) notificationworker stdout | 2023-01-17 05:59:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:43 UTC)" (scheduled at 2023-01-17 05:59:43.935629+00:00) notificationworker stdout | 2023-01-17 05:59:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:59:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:43 UTC)" executed successfully builder stdout | 2023-01-17 05:59:44,649 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:44,658 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:44,658 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:44,664 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:59:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:59:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:59.337413+00:00 (in 11.404448 seconds) namespacegcworker stdout | 2023-01-17 05:59:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:47 UTC)" (scheduled at 2023-01-17 05:59:47.932490+00:00) namespacegcworker stdout | 2023-01-17 05:59:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:59:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:58.735066+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 05:59:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:58 UTC)" (scheduled at 2023-01-17 05:59:48.735066+00:00) notificationworker stdout | 2023-01-17 05:59:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 48, 735905), True, datetime.datetime(2023, 1, 17, 5, 59, 48, 735905), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:59:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:59:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:39.030814+00:00 (in 50.291892 seconds) repositorygcworker stdout | 2023-01-17 05:59:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:48 UTC)" (scheduled at 2023-01-17 05:59:48.738458+00:00) repositorygcworker stdout | 2023-01-17 05:59:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:59:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:59:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:59:48,831 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:24.641816+00:00 (in 35.810805 seconds) exportactionlogsworker stdout | 2023-01-17 05:59:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:48 UTC)" (scheduled at 2023-01-17 05:59:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 05:59:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:59:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 48, 831418), True, datetime.datetime(2023, 1, 17, 5, 59, 48, 831418), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:59:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:59:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:59:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:59:52,626 [239] [DEBUG] [app] Starting request: urn:request:c3daadbf-da4f-406a-b2b9-bdc772e2409c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:52,626 [244] [DEBUG] [app] Starting request: urn:request:66a34a46-4501-4f79-b253-ecf2df304ff0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 05:59:52,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,628 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:52,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:52,631 [232] [DEBUG] [app] Starting request: urn:request:2867ed1c-1445-4ba4-95df-2909544dd1fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,632 [240] [DEBUG] [app] Starting request: urn:request:778eeda5-f918-48c7-9ce9-1cd68802a804 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,632 [232] [DEBUG] [app] Ending request: urn:request:2867ed1c-1445-4ba4-95df-2909544dd1fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2867ed1c-1445-4ba4-95df-2909544dd1fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:59:52,632 [240] [DEBUG] [app] Ending request: urn:request:778eeda5-f918-48c7-9ce9-1cd68802a804 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:778eeda5-f918-48c7-9ce9-1cd68802a804', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:52,633 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:59:52,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:52,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:52,636 [243] [DEBUG] [app] Starting request: urn:request:9fb31fb1-bbc3-4cdd-8203-13614c061917 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:52,637 [241] [DEBUG] [app] Starting request: urn:request:ebc02416-4a9a-4463-b53e-3e30b7a13289 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:52,637 [243] [DEBUG] [app] Ending request: urn:request:9fb31fb1-bbc3-4cdd-8203-13614c061917 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9fb31fb1-bbc3-4cdd-8203-13614c061917', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:59:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:52,637 [241] [DEBUG] [app] Ending request: urn:request:ebc02416-4a9a-4463-b53e-3e30b7a13289 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ebc02416-4a9a-4463-b53e-3e30b7a13289', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:52,649 [239] [DEBUG] [app] Ending request: urn:request:c3daadbf-da4f-406a-b2b9-bdc772e2409c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3daadbf-da4f-406a-b2b9-bdc772e2409c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:52,649 [244] [DEBUG] [app] Ending request: urn:request:66a34a46-4501-4f79-b253-ecf2df304ff0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66a34a46-4501-4f79-b253-ecf2df304ff0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:52,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:59:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 05:59:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:09.230038+00:00 (in 16.392221 seconds) securityworker stdout | 2023-01-17 05:59:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:22 UTC)" (scheduled at 2023-01-17 05:59:52.837365+00:00) securityworker stdout | 2023-01-17 05:59:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:52,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:22 UTC)" executed successfully builder stdout | 2023-01-17 05:59:54,674 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:54,683 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:54,683 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:54,689 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:59:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:59:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:27.226424+00:00 (in 29.999566 seconds) buildlogsarchiver stdout | 2023-01-17 05:59:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:27 UTC)" (scheduled at 2023-01-17 05:59:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 05:59:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 59, 57, 227202), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:59:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:59:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:59:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:08.735066+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:59:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:08 UTC)" (scheduled at 2023-01-17 05:59:58.735066+00:00) notificationworker stdout | 2023-01-17 05:59:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 58, 735952), True, datetime.datetime(2023, 1, 17, 5, 59, 58, 735952), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:59:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:59:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:47.932490+00:00 (in 48.594603 seconds) namespacegcworker stdout | 2023-01-17 05:59:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:59 UTC)" (scheduled at 2023-01-17 05:59:59.337413+00:00) namespacegcworker stdout | 2023-01-17 05:59:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:59:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 59, 338871), True, datetime.datetime(2023, 1, 17, 5, 59, 59, 338871), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:59:59,347 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:59:59,347 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:59:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:00:01,954 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:00:02,221 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:00:02,364 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 06:00:03,252 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:00:03,264 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:00:04,513 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:00:04,699 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:04,708 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:04,708 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:04,714 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:00:05,239 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:00:05,266 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:00:05,633 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:00:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:00:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:35.848675+00:00 (in 29.999438 seconds) gcworker stdout | 2023-01-17 06:00:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:35 UTC)" (scheduled at 2023-01-17 06:00:05.848675+00:00) gcworker stdout | 2023-01-17 06:00:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725605849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:00:05,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:00:05,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:00:05,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:00:06,334 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:00:06,337 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:00:06,339 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:00:06,620 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:00:06,660 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:00:07,500 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 06:00:07,625 [241] [DEBUG] [app] Starting request: urn:request:699eb1cc-c0f1-41b3-96af-2dbc0ecdc8bf (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:07,625 [239] [DEBUG] [app] Starting request: urn:request:66f5b47e-c4fd-4e91-85e9-a8574b59df1e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:07,630 [232] [DEBUG] [app] Starting request: urn:request:09c9b2ba-2071-4243-91a9-47f78d1fef3b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,631 [240] [DEBUG] [app] Starting request: urn:request:f92b9c5c-42d0-4b00-b27f-ffb60a3f3a21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,631 [232] [DEBUG] [app] Ending request: urn:request:09c9b2ba-2071-4243-91a9-47f78d1fef3b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09c9b2ba-2071-4243-91a9-47f78d1fef3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,631 [240] [DEBUG] [app] Ending request: urn:request:f92b9c5c-42d0-4b00-b27f-ffb60a3f3a21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f92b9c5c-42d0-4b00-b27f-ffb60a3f3a21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:00:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:07,636 [244] [DEBUG] [app] Starting request: urn:request:c5225c92-1416-4709-880e-4f95e75588e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:07,636 [243] [DEBUG] [app] Starting request: urn:request:1d25e9d7-b203-4763-8c39-4e7516d904fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:07,636 [244] [DEBUG] [app] Ending request: urn:request:c5225c92-1416-4709-880e-4f95e75588e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5225c92-1416-4709-880e-4f95e75588e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:07,637 [243] [DEBUG] [app] Ending request: urn:request:1d25e9d7-b203-4763-8c39-4e7516d904fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d25e9d7-b203-4763-8c39-4e7516d904fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:00:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:07,648 [239] [DEBUG] [app] Ending request: urn:request:66f5b47e-c4fd-4e91-85e9-a8574b59df1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66f5b47e-c4fd-4e91-85e9-a8574b59df1e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:07,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:00:07,649 [241] [DEBUG] [app] Ending request: urn:request:699eb1cc-c0f1-41b3-96af-2dbc0ecdc8bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:699eb1cc-c0f1-41b3-96af-2dbc0ecdc8bf', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:07,649 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) gunicorn-web stdout | 2023-01-17 06:00:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" queuecleanupworker stdout | 2023-01-17 06:00:08,338 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:00:08,341 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 06:00:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:18.735066+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 06:00:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:18 UTC)" (scheduled at 2023-01-17 06:00:08.735066+00:00) notificationworker stdout | 2023-01-17 06:00:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 8, 735915), True, datetime.datetime(2023, 1, 17, 6, 0, 8, 735915), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:00:08,779 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:00:08,782 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:00:08,784 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:00:08,787 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:00:08,789 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:00:08,794 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:00:08,796 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:00:08,799 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:00:08,801 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:00:08,861 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:00:08,865 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:00:08,869 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:00:08,871 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:00:08,874 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:00:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:22.837365+00:00 (in 13.606843 seconds) securityworker stdout | 2023-01-17 06:00:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:39 UTC)" (scheduled at 2023-01-17 06:00:09.230038+00:00) securityworker stdout | 2023-01-17 06:00:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:00:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:00:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:00:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:00:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:00:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:00:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:00:09,504 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:00:10,116 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:00:10,888 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:00:10,957 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 06:00:10,970 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 06:00:11,069 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:00:14,715 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:14,726 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:14,726 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:14,732 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:28.735066+00:00 (in 9.999578 seconds) notificationworker stdout | 2023-01-17 06:00:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:28 UTC)" (scheduled at 2023-01-17 06:00:18.735066+00:00) notificationworker stdout | 2023-01-17 06:00:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 18, 735881), True, datetime.datetime(2023, 1, 17, 6, 0, 18, 735881), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:00:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:00:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:25.334138+00:00 (in 5.091506 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:00:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:20 UTC)" (scheduled at 2023-01-17 06:00:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:00:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:00:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 20, 243020), True, datetime.datetime(2023, 1, 17, 6, 0, 20, 243020), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:00:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:00:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:00:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:00:22,625 [239] [DEBUG] [app] Starting request: urn:request:db2f0767-9db6-4989-9285-d26240e34fcd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:22,625 [241] [DEBUG] [app] Starting request: urn:request:79d569e7-1fc3-409c-bc1c-d90d2948768e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:22,631 [242] [DEBUG] [app] Starting request: urn:request:e645600d-aa11-47f3-ab06-99da6b8082fb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:22,631 [236] [DEBUG] [app] Starting request: urn:request:74734871-5761-46fe-82be-207e7bbacfc4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:22,631 [242] [DEBUG] [app] Ending request: urn:request:e645600d-aa11-47f3-ab06-99da6b8082fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e645600d-aa11-47f3-ab06-99da6b8082fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:00:22,631 [236] [DEBUG] [app] Ending request: urn:request:74734871-5761-46fe-82be-207e7bbacfc4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:74734871-5761-46fe-82be-207e7bbacfc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:00:22,632 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:00:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:22,635 [244] [DEBUG] [app] Starting request: urn:request:c32f9043-5835-4d5c-8815-f25cddb246aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:22,636 [243] [DEBUG] [app] Starting request: urn:request:e10cd610-ce95-46c5-a110-c6102428fff4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:22,636 [244] [DEBUG] [app] Ending request: urn:request:c32f9043-5835-4d5c-8815-f25cddb246aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c32f9043-5835-4d5c-8815-f25cddb246aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:22,636 [243] [DEBUG] [app] Ending request: urn:request:e10cd610-ce95-46c5-a110-c6102428fff4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e10cd610-ce95-46c5-a110-c6102428fff4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:00:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:22,647 [239] [DEBUG] [app] Ending request: urn:request:db2f0767-9db6-4989-9285-d26240e34fcd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db2f0767-9db6-4989-9285-d26240e34fcd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:22,648 [241] [DEBUG] [app] Ending request: urn:request:79d569e7-1fc3-409c-bc1c-d90d2948768e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79d569e7-1fc3-409c-bc1c-d90d2948768e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:00:22,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 06:00:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:00:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:39.230038+00:00 (in 16.392218 seconds) securityworker stdout | 2023-01-17 06:00:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:52 UTC)" (scheduled at 2023-01-17 06:00:22.837365+00:00) securityworker stdout | 2023-01-17 06:00:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:22,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:00:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:00:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:48.830274+00:00 (in 24.187995 seconds) exportactionlogsworker stdout | 2023-01-17 06:00:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:24 UTC)" (scheduled at 2023-01-17 06:00:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:00:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:00:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:24 UTC)" executed successfully builder stdout | 2023-01-17 06:00:24,734 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:24,743 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:24,743 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:24,749 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:00:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:00:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:20.242198+00:00 (in 54.907605 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:00:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:25 UTC)" (scheduled at 2023-01-17 06:00:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:00:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:00:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:00:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:00:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:57.226424+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 06:00:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:57 UTC)" (scheduled at 2023-01-17 06:00:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:00:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 0, 27, 227253), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:00:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:00:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:00:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:38.735066+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 06:00:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:38 UTC)" (scheduled at 2023-01-17 06:00:28.735066+00:00) notificationworker stdout | 2023-01-17 06:00:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 28, 735911), True, datetime.datetime(2023, 1, 17, 6, 0, 28, 735911), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:00:31,960 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:00:32,229 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:00:32,395 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 06:00:33,271 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:00:33,273 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:00:34,549 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:00:34,759 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:34,768 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:34,768 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:34,774 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:00:35,246 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:00:35,275 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:00:35,668 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:00:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:00:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:05.848675+00:00 (in 29.999491 seconds) gcworker stdout | 2023-01-17 06:00:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:05 UTC)" (scheduled at 2023-01-17 06:00:35.848675+00:00) gcworker stdout | 2023-01-17 06:00:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725635849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:00:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:00:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:00:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:00:36,342 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:00:36,345 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:00:36,347 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:00:36,656 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:00:36,695 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:00:37,530 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 06:00:37,625 [241] [DEBUG] [app] Starting request: urn:request:960c1595-b28c-4fa0-8840-2de083c2501d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:37,626 [239] [DEBUG] [app] Starting request: urn:request:ff59004d-d409-43d4-af65-1358fdd7c8d3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:37,630 [240] [DEBUG] [app] Starting request: urn:request:4cf712c5-2739-4856-a0bf-0a6550daf1c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:37,631 [232] [DEBUG] [app] Starting request: urn:request:7b91152a-017f-4eb5-95fa-a32b95374f80 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:37,631 [240] [DEBUG] [app] Ending request: urn:request:4cf712c5-2739-4856-a0bf-0a6550daf1c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4cf712c5-2739-4856-a0bf-0a6550daf1c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:00:37,631 [232] [DEBUG] [app] Ending request: urn:request:7b91152a-017f-4eb5-95fa-a32b95374f80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b91152a-017f-4eb5-95fa-a32b95374f80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:00:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:00:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:37,635 [244] [DEBUG] [app] Starting request: urn:request:ed72d863-7073-487d-8dec-6df24954fb2c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:37,636 [243] [DEBUG] [app] Starting request: urn:request:e0811248-c29a-4e3a-98f9-3a54578405b2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:37,636 [244] [DEBUG] [app] Ending request: urn:request:ed72d863-7073-487d-8dec-6df24954fb2c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed72d863-7073-487d-8dec-6df24954fb2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:37,636 [243] [DEBUG] [app] Ending request: urn:request:e0811248-c29a-4e3a-98f9-3a54578405b2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0811248-c29a-4e3a-98f9-3a54578405b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:00:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:37,646 [241] [DEBUG] [app] Ending request: urn:request:960c1595-b28c-4fa0-8840-2de083c2501d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:960c1595-b28c-4fa0-8840-2de083c2501d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:37,647 [239] [DEBUG] [app] Ending request: urn:request:ff59004d-d409-43d4-af65-1358fdd7c8d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff59004d-d409-43d4-af65-1358fdd7c8d3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 06:00:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) queuecleanupworker stdout | 2023-01-17 06:00:38,349 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:00:38,356 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 06:00:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:43.935629+00:00 (in 5.200120 seconds) notificationworker stdout | 2023-01-17 06:00:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:48 UTC)" (scheduled at 2023-01-17 06:00:38.735066+00:00) notificationworker stdout | 2023-01-17 06:00:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 38, 735899), True, datetime.datetime(2023, 1, 17, 6, 0, 38, 735899), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:00:38,788 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:00:38,790 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:00:38,792 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:00:38,795 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:00:38,797 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:00:38,801 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:00:38,804 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:00:38,806 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:00:38,812 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:00:38,867 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:00:38,871 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:00:38,875 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:00:38,878 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:00:38,880 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:00:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:00:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:48.738458+00:00 (in 9.707238 seconds) repositorygcworker stdout | 2023-01-17 06:00:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:39 UTC)" (scheduled at 2023-01-17 06:00:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:00:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:00:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 39, 31763), True, datetime.datetime(2023, 1, 17, 6, 0, 39, 31763), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:00:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:00:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:00:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:00:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:52.837365+00:00 (in 13.606904 seconds) securityworker stdout | 2023-01-17 06:00:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:09 UTC)" (scheduled at 2023-01-17 06:00:39.230038+00:00) securityworker stdout | 2023-01-17 06:00:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:00:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:00:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:00:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:00:39,525 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:00:40,151 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:00:40,924 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:00:40,992 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 06:00:41,005 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 06:00:41,103 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:00:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:48.735066+00:00 (in 4.798916 seconds) notificationworker stdout | 2023-01-17 06:00:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:43 UTC)" (scheduled at 2023-01-17 06:00:43.935629+00:00) notificationworker stdout | 2023-01-17 06:00:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:00:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:43 UTC)" executed successfully builder stdout | 2023-01-17 06:00:44,783 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:44,792 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:44,792 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:44,798 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:00:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:00:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:59.337413+00:00 (in 11.404488 seconds) namespacegcworker stdout | 2023-01-17 06:00:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:47 UTC)" (scheduled at 2023-01-17 06:00:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:00:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:00:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:58.735066+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 06:00:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:58 UTC)" (scheduled at 2023-01-17 06:00:48.735066+00:00) notificationworker stdout | 2023-01-17 06:00:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 48, 735917), True, datetime.datetime(2023, 1, 17, 6, 0, 48, 735917), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:00:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:00:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:39.030814+00:00 (in 50.291914 seconds) repositorygcworker stdout | 2023-01-17 06:00:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:48 UTC)" (scheduled at 2023-01-17 06:00:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:00:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:00:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:00:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:00:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:24.641816+00:00 (in 35.811053 seconds) exportactionlogsworker stdout | 2023-01-17 06:00:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:48 UTC)" (scheduled at 2023-01-17 06:00:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:00:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:00:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 48, 831128), True, datetime.datetime(2023, 1, 17, 6, 0, 48, 831128), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:00:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:00:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:00:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:00:52,625 [243] [DEBUG] [app] Starting request: urn:request:d34fd02e-fd48-4b53-8862-6258c182c702 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:52,625 [244] [DEBUG] [app] Starting request: urn:request:d46616b3-49e3-418b-858c-9ecf7960976a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:00:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:52,630 [232] [DEBUG] [app] Starting request: urn:request:cb6e94fd-8a18-480e-ab6d-51bba4936823 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,630 [236] [DEBUG] [app] Starting request: urn:request:3c093709-8f2c-4c27-b0c8-d6a2c8c84ad5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,631 [232] [DEBUG] [app] Ending request: urn:request:cb6e94fd-8a18-480e-ab6d-51bba4936823 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb6e94fd-8a18-480e-ab6d-51bba4936823', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:00:52,631 [236] [DEBUG] [app] Ending request: urn:request:3c093709-8f2c-4c27-b0c8-d6a2c8c84ad5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c093709-8f2c-4c27-b0c8-d6a2c8c84ad5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:00:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:00:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:52,635 [239] [DEBUG] [app] Starting request: urn:request:9b4faa75-db71-4060-9dd2-b70129010de2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:52,636 [243] [DEBUG] [app] Starting request: urn:request:2eb63565-fe9a-482e-9fdf-99ad8644df7c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:52,636 [239] [DEBUG] [app] Ending request: urn:request:9b4faa75-db71-4060-9dd2-b70129010de2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b4faa75-db71-4060-9dd2-b70129010de2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:00:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:52,636 [243] [DEBUG] [app] Ending request: urn:request:2eb63565-fe9a-482e-9fdf-99ad8644df7c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2eb63565-fe9a-482e-9fdf-99ad8644df7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:52,647 [243] [DEBUG] [app] Ending request: urn:request:d34fd02e-fd48-4b53-8862-6258c182c702 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d34fd02e-fd48-4b53-8862-6258c182c702', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:52,647 [244] [DEBUG] [app] Ending request: urn:request:d46616b3-49e3-418b-858c-9ecf7960976a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d46616b3-49e3-418b-858c-9ecf7960976a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 06:00:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 06:00:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:09.230038+00:00 (in 16.392282 seconds) securityworker stdout | 2023-01-17 06:00:52,837 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:22 UTC)" (scheduled at 2023-01-17 06:00:52.837365+00:00) securityworker stdout | 2023-01-17 06:00:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:22 UTC)" executed successfully builder stdout | 2023-01-17 06:00:54,808 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:54,817 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:54,817 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:54,823 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:00:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:00:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:27.226424+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 06:00:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:27 UTC)" (scheduled at 2023-01-17 06:00:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:00:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 0, 57, 227521), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:00:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:00:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:00:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:08.735066+00:00 (in 9.999508 seconds) notificationworker stdout | 2023-01-17 06:00:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:08 UTC)" (scheduled at 2023-01-17 06:00:58.735066+00:00) notificationworker stdout | 2023-01-17 06:00:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 58, 735994), True, datetime.datetime(2023, 1, 17, 6, 0, 58, 735994), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:00:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:00:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:47.932490+00:00 (in 48.594596 seconds) namespacegcworker stdout | 2023-01-17 06:00:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:59 UTC)" (scheduled at 2023-01-17 06:00:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:00:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:00:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 59, 338341), True, datetime.datetime(2023, 1, 17, 6, 0, 59, 338341), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:00:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:00:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:00:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:01:01,982 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:01:02,250 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:01:02,428 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:01:03,306 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:01:03,308 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:01:04,561 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:01:04,833 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:04,841 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:04,841 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:04,847 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:01:05,282 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:01:05,285 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:01:05,703 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:01:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:01:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:35.848675+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 06:01:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:35 UTC)" (scheduled at 2023-01-17 06:01:05.848675+00:00) gcworker stdout | 2023-01-17 06:01:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725665849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:01:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:01:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:01:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:01:06,349 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:01:06,352 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:01:06,354 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:01:06,663 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:01:06,730 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:01:07,540 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 06:01:07,625 [239] [DEBUG] [app] Starting request: urn:request:e0c7814e-af9f-4138-8a4b-03a64eb4f675 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:07,625 [243] [DEBUG] [app] Starting request: urn:request:f392f5db-7c24-4a45-a7f0-016dbd93ddf3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:07,630 [232] [DEBUG] [app] Starting request: urn:request:35e58215-56aa-458f-808d-544f13810a7d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:07,630 [237] [DEBUG] [app] Starting request: urn:request:5066c1e0-f1d8-41fc-8e91-c28df3428d34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:07,630 [232] [DEBUG] [app] Ending request: urn:request:35e58215-56aa-458f-808d-544f13810a7d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35e58215-56aa-458f-808d-544f13810a7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:01:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:01:07,631 [237] [DEBUG] [app] Ending request: urn:request:5066c1e0-f1d8-41fc-8e91-c28df3428d34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5066c1e0-f1d8-41fc-8e91-c28df3428d34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:01:07,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:07,635 [241] [DEBUG] [app] Starting request: urn:request:166fad46-b4a3-47ee-b5b6-5d33ddf9cdc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:07,635 [241] [DEBUG] [app] Ending request: urn:request:166fad46-b4a3-47ee-b5b6-5d33ddf9cdc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:166fad46-b4a3-47ee-b5b6-5d33ddf9cdc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:07,636 [244] [DEBUG] [app] Starting request: urn:request:c03c2033-82a8-40bc-a163-dae4baee15e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:01:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:07,636 [244] [DEBUG] [app] Ending request: urn:request:c03c2033-82a8-40bc-a163-dae4baee15e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c03c2033-82a8-40bc-a163-dae4baee15e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:07,647 [239] [DEBUG] [app] Ending request: urn:request:e0c7814e-af9f-4138-8a4b-03a64eb4f675 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0c7814e-af9f-4138-8a4b-03a64eb4f675', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:07,647 [243] [DEBUG] [app] Ending request: urn:request:f392f5db-7c24-4a45-a7f0-016dbd93ddf3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f392f5db-7c24-4a45-a7f0-016dbd93ddf3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:07,647 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:01:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:01:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" queuecleanupworker stdout | 2023-01-17 06:01:08,375 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:01:08,391 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 06:01:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:18.735066+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 06:01:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:18 UTC)" (scheduled at 2023-01-17 06:01:08.735066+00:00) notificationworker stdout | 2023-01-17 06:01:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 8, 735875), True, datetime.datetime(2023, 1, 17, 6, 1, 8, 735875), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:01:08,796 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:01:08,798 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:01:08,801 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:01:08,803 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:01:08,806 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:01:08,809 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:01:08,811 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:01:08,815 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:01:08,821 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:01:08,873 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:01:08,878 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:01:08,881 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:01:08,885 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:01:08,888 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:01:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:22.837365+00:00 (in 13.606852 seconds) securityworker stdout | 2023-01-17 06:01:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:39 UTC)" (scheduled at 2023-01-17 06:01:09.230038+00:00) securityworker stdout | 2023-01-17 06:01:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:01:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:01:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:01:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:01:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:01:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:01:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:01:09,542 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:01:10,166 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:01:10,960 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:01:11,012 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:01:11,027 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:01:11,139 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:01:14,857 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:14,866 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:14,866 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:14,872 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:28.735066+00:00 (in 9.999589 seconds) notificationworker stdout | 2023-01-17 06:01:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:28 UTC)" (scheduled at 2023-01-17 06:01:18.735066+00:00) notificationworker stdout | 2023-01-17 06:01:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 18, 735876), True, datetime.datetime(2023, 1, 17, 6, 1, 18, 735876), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:01:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:01:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:25.334138+00:00 (in 5.091493 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:01:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:20 UTC)" (scheduled at 2023-01-17 06:01:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:01:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:01:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 20, 243044), True, datetime.datetime(2023, 1, 17, 6, 1, 20, 243044), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:01:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:01:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:01:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:01:22,625 [244] [DEBUG] [app] Starting request: urn:request:8c055e60-024b-4866-8bf8-21c4882702ef (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:22,625 [239] [DEBUG] [app] Starting request: urn:request:eaa563ec-9df9-4a94-8c69-9d5444592526 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:22,629 [232] [DEBUG] [app] Starting request: urn:request:c5383b33-4916-4f75-947a-b21f60ce8e01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:22,630 [232] [DEBUG] [app] Ending request: urn:request:c5383b33-4916-4f75-947a-b21f60ce8e01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5383b33-4916-4f75-947a-b21f60ce8e01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:01:22,630 [242] [DEBUG] [app] Starting request: urn:request:ce24ea40-df1f-409d-b50b-ebd084233919 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:22,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:01:22,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:01:22,631 [242] [DEBUG] [app] Ending request: urn:request:ce24ea40-df1f-409d-b50b-ebd084233919 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ce24ea40-df1f-409d-b50b-ebd084233919', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:01:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,633 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:22,634 [239] [DEBUG] [app] Starting request: urn:request:8252f876-fe58-4437-99f5-d5ccaef7ce96 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:22,635 [239] [DEBUG] [app] Ending request: urn:request:8252f876-fe58-4437-99f5-d5ccaef7ce96 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8252f876-fe58-4437-99f5-d5ccaef7ce96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:22,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:22,635 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:22,635 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:22,636 [241] [DEBUG] [app] Starting request: urn:request:52ca5b3c-7c6e-4a86-8a6e-dc421339343d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:22,637 [241] [DEBUG] [app] Ending request: urn:request:52ca5b3c-7c6e-4a86-8a6e-dc421339343d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:52ca5b3c-7c6e-4a86-8a6e-dc421339343d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:22,640 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:22,646 [244] [DEBUG] [app] Ending request: urn:request:8c055e60-024b-4866-8bf8-21c4882702ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8c055e60-024b-4866-8bf8-21c4882702ef', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:22,646 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 06:01:22,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:01:22,647 [239] [DEBUG] [app] Ending request: urn:request:eaa563ec-9df9-4a94-8c69-9d5444592526 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eaa563ec-9df9-4a94-8c69-9d5444592526', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:22,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:01:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:01:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:39.230038+00:00 (in 16.392149 seconds) securityworker stdout | 2023-01-17 06:01:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:52 UTC)" (scheduled at 2023-01-17 06:01:22.837365+00:00) securityworker stdout | 2023-01-17 06:01:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:01:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:01:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:44.641497+00:00 (in 19.999267 seconds) exportactionlogsworker stdout | 2023-01-17 06:01:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:24 UTC)" (scheduled at 2023-01-17 06:01:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:01:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:01:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:24 UTC)" executed successfully builder stdout | 2023-01-17 06:01:24,875 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:24,884 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:24,884 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:24,890 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:01:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:01:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:39.333860+00:00 (in 13.999307 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:01:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:25 UTC)" (scheduled at 2023-01-17 06:01:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:01:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:01:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:01:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:01:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:57.226424+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 06:01:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:57 UTC)" (scheduled at 2023-01-17 06:01:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:01:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 1, 27, 227264), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:01:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:01:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:01:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:38.735066+00:00 (in 9.999537 seconds) notificationworker stdout | 2023-01-17 06:01:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:38 UTC)" (scheduled at 2023-01-17 06:01:28.735066+00:00) notificationworker stdout | 2023-01-17 06:01:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 28, 736038), True, datetime.datetime(2023, 1, 17, 6, 1, 28, 736038), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:01:32,002 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:01:32,256 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:01:32,460 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 06:01:33,317 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:01:33,341 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:01:34,580 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:01:34,900 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:34,909 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:34,909 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:34,915 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:01:35,291 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:01:35,316 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:01:35,737 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:01:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:01:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:05.848675+00:00 (in 29.999575 seconds) gcworker stdout | 2023-01-17 06:01:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:05 UTC)" (scheduled at 2023-01-17 06:01:35.848675+00:00) gcworker stdout | 2023-01-17 06:01:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725695849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:01:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:01:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:01:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:01:36,357 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:01:36,359 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:01:36,361 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:01:36,679 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:01:36,741 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:01:37,574 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 06:01:37,625 [239] [DEBUG] [app] Starting request: urn:request:7661b1ff-ffe6-41e7-8e0a-2c644e3717d9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:37,626 [244] [DEBUG] [app] Starting request: urn:request:3d9d5d6e-b2e4-4820-a84e-3458909a81a9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:37,630 [237] [DEBUG] [app] Starting request: urn:request:49f21417-700b-4f9d-a310-990029371a04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:37,631 [240] [DEBUG] [app] Starting request: urn:request:1e77af40-aa61-4acf-8765-f1527538d3ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:37,631 [237] [DEBUG] [app] Ending request: urn:request:49f21417-700b-4f9d-a310-990029371a04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49f21417-700b-4f9d-a310-990029371a04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:01:37,631 [240] [DEBUG] [app] Ending request: urn:request:1e77af40-aa61-4acf-8765-f1527538d3ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e77af40-aa61-4acf-8765-f1527538d3ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:01:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:01:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:37,635 [244] [DEBUG] [app] Starting request: urn:request:4cf630e0-d624-4294-a7af-a6e48c418ffd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:37,636 [244] [DEBUG] [app] Ending request: urn:request:4cf630e0-d624-4294-a7af-a6e48c418ffd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4cf630e0-d624-4294-a7af-a6e48c418ffd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:01:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:37,637 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:37,637 [243] [DEBUG] [app] Starting request: urn:request:60c684c8-f17e-4d4c-8dfb-5e73196d9e00 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:37,638 [243] [DEBUG] [app] Ending request: urn:request:60c684c8-f17e-4d4c-8dfb-5e73196d9e00 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:60c684c8-f17e-4d4c-8dfb-5e73196d9e00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:37,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:37,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:37,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:37,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:37,647 [239] [DEBUG] [app] Ending request: urn:request:7661b1ff-ffe6-41e7-8e0a-2c644e3717d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7661b1ff-ffe6-41e7-8e0a-2c644e3717d9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:37,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:01:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:01:37,648 [244] [DEBUG] [app] Ending request: urn:request:3d9d5d6e-b2e4-4820-a84e-3458909a81a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d9d5d6e-b2e4-4820-a84e-3458909a81a9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:37,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:01:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" queuecleanupworker stdout | 2023-01-17 06:01:38,411 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:01:38,423 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 06:01:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:43.935629+00:00 (in 5.200139 seconds) notificationworker stdout | 2023-01-17 06:01:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:48 UTC)" (scheduled at 2023-01-17 06:01:38.735066+00:00) notificationworker stdout | 2023-01-17 06:01:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 38, 735847), True, datetime.datetime(2023, 1, 17, 6, 1, 38, 735847), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:01:38,804 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:01:38,807 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:01:38,809 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:01:38,814 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:01:38,817 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:01:38,819 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:01:38,821 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:01:38,824 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:01:38,830 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:01:38,879 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:01:38,884 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:01:38,887 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:01:38,892 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:01:38,894 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:01:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:01:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:48.738458+00:00 (in 9.707221 seconds) repositorygcworker stdout | 2023-01-17 06:01:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:39 UTC)" (scheduled at 2023-01-17 06:01:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:01:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:01:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 39, 31670), True, datetime.datetime(2023, 1, 17, 6, 1, 39, 31670), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:01:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:01:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:01:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:01:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:52.837365+00:00 (in 13.606883 seconds) securityworker stdout | 2023-01-17 06:01:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:09 UTC)" (scheduled at 2023-01-17 06:01:39.230038+00:00) securityworker stdout | 2023-01-17 06:01:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:01:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:01:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:39,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:39,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:01:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:01:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:01:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:01:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:01:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:01:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:20.242198+00:00 (in 40.907944 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:01:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:39 UTC)" (scheduled at 2023-01-17 06:01:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:01:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 1, 39, 334611), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:01:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 39, 334611), True, datetime.datetime(2023, 1, 17, 6, 1, 39, 334611), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:01:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 39, 334611), True, datetime.datetime(2023, 1, 17, 6, 1, 39, 334611), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 1, 39, 334611), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:01:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:01:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:01:39,567 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:01:40,201 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:01:40,967 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:01:41,027 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:01:41,048 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:01:41,174 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:01:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:48.735066+00:00 (in 4.798958 seconds) notificationworker stdout | 2023-01-17 06:01:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:43 UTC)" (scheduled at 2023-01-17 06:01:43.935629+00:00) notificationworker stdout | 2023-01-17 06:01:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:01:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:01:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:01:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:48.830274+00:00 (in 4.188379 seconds) exportactionlogsworker stdout | 2023-01-17 06:01:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:44 UTC)" (scheduled at 2023-01-17 06:01:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 06:01:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 1, 44, 642238), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:01:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 44, 642238), True, datetime.datetime(2023, 1, 17, 6, 1, 44, 642238), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:01:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 44, 642238), True, datetime.datetime(2023, 1, 17, 6, 1, 44, 642238), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 1, 44, 642238), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:01:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:01:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:44 UTC)" executed successfully builder stdout | 2023-01-17 06:01:44,918 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:44,926 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:44,926 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:44,932 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:01:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:01:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:59.337413+00:00 (in 11.404428 seconds) namespacegcworker stdout | 2023-01-17 06:01:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:47 UTC)" (scheduled at 2023-01-17 06:01:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:01:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:01:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:58.735066+00:00 (in 9.999501 seconds) notificationworker stdout | 2023-01-17 06:01:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:58 UTC)" (scheduled at 2023-01-17 06:01:48.735066+00:00) notificationworker stdout | 2023-01-17 06:01:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 48, 735940), True, datetime.datetime(2023, 1, 17, 6, 1, 48, 735940), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:01:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:01:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:04.738062+00:00 (in 15.999144 seconds) repositorygcworker stdout | 2023-01-17 06:01:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:48 UTC)" (scheduled at 2023-01-17 06:01:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:01:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:01:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:01:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:01:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:24.641816+00:00 (in 35.811137 seconds) exportactionlogsworker stdout | 2023-01-17 06:01:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:48 UTC)" (scheduled at 2023-01-17 06:01:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:01:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:01:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 48, 831063), True, datetime.datetime(2023, 1, 17, 6, 1, 48, 831063), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:01:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:01:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:01:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:01:52,625 [239] [DEBUG] [app] Starting request: urn:request:d687753d-e640-4cce-a864-fd3a54b6254d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:52,626 [244] [DEBUG] [app] Starting request: urn:request:648d7ae1-3ae0-42e2-95a1-7570ecba5ed5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:01:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:52,631 [232] [DEBUG] [app] Starting request: urn:request:448e6a56-c82e-42ba-915f-a9f8dc43064a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:52,631 [240] [DEBUG] [app] Starting request: urn:request:3539d01a-2b94-4fd5-9367-a8ca582a4d5a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:52,631 [232] [DEBUG] [app] Ending request: urn:request:448e6a56-c82e-42ba-915f-a9f8dc43064a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:448e6a56-c82e-42ba-915f-a9f8dc43064a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:01:52,631 [240] [DEBUG] [app] Ending request: urn:request:3539d01a-2b94-4fd5-9367-a8ca582a4d5a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3539d01a-2b94-4fd5-9367-a8ca582a4d5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:01:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:01:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:01:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:52,636 [243] [DEBUG] [app] Starting request: urn:request:af372a96-91fe-47dc-b3b1-0ccea7c7bf93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:52,636 [241] [DEBUG] [app] Starting request: urn:request:830e3da3-0256-4afa-a7c1-7b8447c2900f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:52,637 [241] [DEBUG] [app] Ending request: urn:request:830e3da3-0256-4afa-a7c1-7b8447c2900f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:830e3da3-0256-4afa-a7c1-7b8447c2900f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:52,637 [243] [DEBUG] [app] Ending request: urn:request:af372a96-91fe-47dc-b3b1-0ccea7c7bf93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af372a96-91fe-47dc-b3b1-0ccea7c7bf93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:52,648 [244] [DEBUG] [app] Ending request: urn:request:648d7ae1-3ae0-42e2-95a1-7570ecba5ed5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:648d7ae1-3ae0-42e2-95a1-7570ecba5ed5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:52,648 [239] [DEBUG] [app] Ending request: urn:request:d687753d-e640-4cce-a864-fd3a54b6254d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d687753d-e640-4cce-a864-fd3a54b6254d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:01:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:01:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:09.230038+00:00 (in 16.392244 seconds) securityworker stdout | 2023-01-17 06:01:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:22 UTC)" (scheduled at 2023-01-17 06:01:52.837365+00:00) securityworker stdout | 2023-01-17 06:01:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:22 UTC)" executed successfully builder stdout | 2023-01-17 06:01:54,942 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:54,951 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:54,951 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:54,957 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:01:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:01:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:27.226424+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 06:01:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:27 UTC)" (scheduled at 2023-01-17 06:01:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:01:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 1, 57, 227232), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:01:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:01:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:01:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:08.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 06:01:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:08 UTC)" (scheduled at 2023-01-17 06:01:58.735066+00:00) notificationworker stdout | 2023-01-17 06:01:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 58, 735965), True, datetime.datetime(2023, 1, 17, 6, 1, 58, 735965), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:01:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:01:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:47.932490+00:00 (in 48.594624 seconds) namespacegcworker stdout | 2023-01-17 06:01:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:59 UTC)" (scheduled at 2023-01-17 06:01:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:01:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:01:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 59, 338260), True, datetime.datetime(2023, 1, 17, 6, 1, 59, 338260), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:01:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:01:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:01:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:02:02,038 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:02:02,291 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:02:02,471 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 06:02:03,344 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:02:03,358 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:02:04,586 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} repositorygcworker stdout | 2023-01-17 06:02:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:02:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:39.030814+00:00 (in 34.292359 seconds) repositorygcworker stdout | 2023-01-17 06:02:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:07:04 UTC)" (scheduled at 2023-01-17 06:02:04.738062+00:00) repositorygcworker stdout | 2023-01-17 06:02:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 2, 4, 738823), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:02:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 2, 4, 738823), True, datetime.datetime(2023, 1, 17, 6, 2, 4, 738823), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:02:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 2, 4, 738823), True, datetime.datetime(2023, 1, 17, 6, 2, 4, 738823), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 2, 4, 738823), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:02:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:02:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:07:04 UTC)" executed successfully builder stdout | 2023-01-17 06:02:04,967 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:04,975 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:04,975 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:04,981 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:02:05,327 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:02:05,351 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:02:05,773 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:02:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:02:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:35.848675+00:00 (in 29.999573 seconds) gcworker stdout | 2023-01-17 06:02:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:35 UTC)" (scheduled at 2023-01-17 06:02:05.848675+00:00) gcworker stdout | 2023-01-17 06:02:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725725849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:02:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:02:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:02:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:02:06,363 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:02:06,366 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:02:06,368 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:02:06,708 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:02:06,776 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:02:07,585 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 06:02:07,626 [243] [DEBUG] [app] Starting request: urn:request:a726c48b-5d97-465f-9a0e-a1d726fa5494 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:07,626 [239] [DEBUG] [app] Starting request: urn:request:3c97326e-640d-4aec-bb2a-f68218809f03 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:07,630 [232] [DEBUG] [app] Starting request: urn:request:415f728b-f3c6-4587-93c7-962d099b9c1a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:07,631 [237] [DEBUG] [app] Starting request: urn:request:9662d435-4832-47b9-821b-940b4926a099 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:07,631 [232] [DEBUG] [app] Ending request: urn:request:415f728b-f3c6-4587-93c7-962d099b9c1a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:415f728b-f3c6-4587-93c7-962d099b9c1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:02:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:02:07,631 [237] [DEBUG] [app] Ending request: urn:request:9662d435-4832-47b9-821b-940b4926a099 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9662d435-4832-47b9-821b-940b4926a099', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:02:07,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:02:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:07,635 [241] [DEBUG] [app] Starting request: urn:request:b7b97d4b-2ce7-4684-a95f-15d78d6c93c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:07,635 [241] [DEBUG] [app] Ending request: urn:request:b7b97d4b-2ce7-4684-a95f-15d78d6c93c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7b97d4b-2ce7-4684-a95f-15d78d6c93c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:07,636 [244] [DEBUG] [app] Starting request: urn:request:82a664fc-f246-4d9a-bd30-4e98c2152958 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:07,636 [244] [DEBUG] [app] Ending request: urn:request:82a664fc-f246-4d9a-bd30-4e98c2152958 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82a664fc-f246-4d9a-bd30-4e98c2152958', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:02:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:07,646 [243] [DEBUG] [app] Ending request: urn:request:a726c48b-5d97-465f-9a0e-a1d726fa5494 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a726c48b-5d97-465f-9a0e-a1d726fa5494', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:07,647 [239] [DEBUG] [app] Ending request: urn:request:3c97326e-640d-4aec-bb2a-f68218809f03 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c97326e-640d-4aec-bb2a-f68218809f03', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:02:07,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 06:02:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" queuecleanupworker stdout | 2023-01-17 06:02:08,447 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:02:08,449 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 06:02:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:18.735066+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 06:02:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:18 UTC)" (scheduled at 2023-01-17 06:02:08.735066+00:00) notificationworker stdout | 2023-01-17 06:02:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 8, 735864), True, datetime.datetime(2023, 1, 17, 6, 2, 8, 735864), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:02:08,812 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:02:08,815 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:02:08,819 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:02:08,824 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:02:08,827 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:02:08,829 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:02:08,832 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:02:08,834 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:02:08,839 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:02:08,885 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:02:08,890 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:02:08,893 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:02:08,897 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:02:08,900 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:02:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:22.837365+00:00 (in 13.606871 seconds) securityworker stdout | 2023-01-17 06:02:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:39 UTC)" (scheduled at 2023-01-17 06:02:09.230038+00:00) securityworker stdout | 2023-01-17 06:02:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:02:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:02:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:02:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:02:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:02:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:02:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:02:09,603 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:02:10,237 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:02:10,999 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:02:11,061 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:02:11,083 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:02:11,209 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:02:14,991 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:15,000 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:15,001 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:15,006 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:28.735066+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 06:02:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:28 UTC)" (scheduled at 2023-01-17 06:02:18.735066+00:00) notificationworker stdout | 2023-01-17 06:02:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 18, 735940), True, datetime.datetime(2023, 1, 17, 6, 2, 18, 735940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:02:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:02:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:25.334138+00:00 (in 5.091492 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:02:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:20 UTC)" (scheduled at 2023-01-17 06:02:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:02:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:02:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 20, 243475), True, datetime.datetime(2023, 1, 17, 6, 2, 20, 243475), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:02:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:02:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:02:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:02:22,625 [243] [DEBUG] [app] Starting request: urn:request:2e4b5202-1431-45b2-863d-74f0b156a2ac (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:22,625 [239] [DEBUG] [app] Starting request: urn:request:4211c938-b216-4118-9efd-ab2b552abe44 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:22,630 [242] [DEBUG] [app] Starting request: urn:request:7ac3fd7d-be4d-4e25-9916-7bf2109b0503 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,631 [240] [DEBUG] [app] Starting request: urn:request:8cda5039-7faa-4426-bada-e495eac6c737 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,631 [242] [DEBUG] [app] Ending request: urn:request:7ac3fd7d-be4d-4e25-9916-7bf2109b0503 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ac3fd7d-be4d-4e25-9916-7bf2109b0503', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,631 [240] [DEBUG] [app] Ending request: urn:request:8cda5039-7faa-4426-bada-e495eac6c737 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8cda5039-7faa-4426-bada-e495eac6c737', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:02:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:02:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:22,636 [241] [DEBUG] [app] Starting request: urn:request:151d99fe-e10e-4733-8e94-c623f44ce67d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:22,636 [243] [DEBUG] [app] Starting request: urn:request:c420849c-723d-479b-8379-f46a7a4d2db7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:22,636 [241] [DEBUG] [app] Ending request: urn:request:151d99fe-e10e-4733-8e94-c623f44ce67d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:151d99fe-e10e-4733-8e94-c623f44ce67d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:22,637 [243] [DEBUG] [app] Ending request: urn:request:c420849c-723d-479b-8379-f46a7a4d2db7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c420849c-723d-479b-8379-f46a7a4d2db7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:22,648 [243] [DEBUG] [app] Ending request: urn:request:2e4b5202-1431-45b2-863d-74f0b156a2ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e4b5202-1431-45b2-863d-74f0b156a2ac', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:22,648 [239] [DEBUG] [app] Ending request: urn:request:4211c938-b216-4118-9efd-ab2b552abe44 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4211c938-b216-4118-9efd-ab2b552abe44', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:02:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:02:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:39.230038+00:00 (in 16.392184 seconds) securityworker stdout | 2023-01-17 06:02:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:52 UTC)" (scheduled at 2023-01-17 06:02:22.837365+00:00) securityworker stdout | 2023-01-17 06:02:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:02:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:02:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:48.830274+00:00 (in 24.187993 seconds) exportactionlogsworker stdout | 2023-01-17 06:02:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:24 UTC)" (scheduled at 2023-01-17 06:02:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:02:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:02:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:24 UTC)" executed successfully builder stdout | 2023-01-17 06:02:25,016 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:25,025 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:25,025 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:25,031 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:02:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:02:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:20.242198+00:00 (in 54.907607 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:02:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:25 UTC)" (scheduled at 2023-01-17 06:02:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:02:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:02:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:02:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:02:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:57.226424+00:00 (in 29.999570 seconds) buildlogsarchiver stdout | 2023-01-17 06:02:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:57 UTC)" (scheduled at 2023-01-17 06:02:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:02:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 2, 27, 227224), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:02:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:02:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:02:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:38.735066+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 06:02:28,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:38 UTC)" (scheduled at 2023-01-17 06:02:28.735066+00:00) notificationworker stdout | 2023-01-17 06:02:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 28, 736369), True, datetime.datetime(2023, 1, 17, 6, 2, 28, 736369), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:02:32,074 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:02:32,300 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:02:32,506 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 06:02:33,378 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:02:33,396 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:02:34,622 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:02:35,041 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:35,050 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:35,050 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:35,055 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:02:35,341 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:02:35,386 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:02:35,785 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:02:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:02:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:05.848675+00:00 (in 29.999579 seconds) gcworker stdout | 2023-01-17 06:02:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:05 UTC)" (scheduled at 2023-01-17 06:02:35.848675+00:00) gcworker stdout | 2023-01-17 06:02:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725755849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:02:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:02:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:02:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:02:36,371 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:02:36,374 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:02:36,376 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:02:36,718 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:02:36,812 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:02:37,621 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 06:02:37,625 [243] [DEBUG] [app] Starting request: urn:request:1a132335-7501-4a7b-8f79-a094aacd50d4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:37,625 [244] [DEBUG] [app] Starting request: urn:request:4c1517b8-a97e-4c5c-87e3-7ed9134dbda2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:37,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:37,630 [232] [DEBUG] [app] Starting request: urn:request:1b5ab571-fb47-4f35-b286-92530d9f7948 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:37,630 [242] [DEBUG] [app] Starting request: urn:request:08681eb9-e3e6-42e6-817d-0eba36aad2a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:37,630 [232] [DEBUG] [app] Ending request: urn:request:1b5ab571-fb47-4f35-b286-92530d9f7948 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b5ab571-fb47-4f35-b286-92530d9f7948', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:37,630 [242] [DEBUG] [app] Ending request: urn:request:08681eb9-e3e6-42e6-817d-0eba36aad2a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08681eb9-e3e6-42e6-817d-0eba36aad2a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:02:37,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:02:37,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:37,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:02:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,633 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:37,634 [239] [DEBUG] [app] Starting request: urn:request:98a4d327-2a0c-488e-985c-340507ab513b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:37,635 [241] [DEBUG] [app] Starting request: urn:request:d0afabe9-bd2d-4696-b8f4-955d7ce68357 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:37,635 [239] [DEBUG] [app] Ending request: urn:request:98a4d327-2a0c-488e-985c-340507ab513b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98a4d327-2a0c-488e-985c-340507ab513b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:02:37,635 [241] [DEBUG] [app] Ending request: urn:request:d0afabe9-bd2d-4696-b8f4-955d7ce68357 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d0afabe9-bd2d-4696-b8f4-955d7ce68357', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:37,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:37,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:37,635 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:37,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:37,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:37,640 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:37,645 [243] [DEBUG] [app] Ending request: urn:request:1a132335-7501-4a7b-8f79-a094aacd50d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1a132335-7501-4a7b-8f79-a094aacd50d4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:37,645 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:37,646 [244] [DEBUG] [app] Ending request: urn:request:4c1517b8-a97e-4c5c-87e3-7ed9134dbda2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c1517b8-a97e-4c5c-87e3-7ed9134dbda2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:37,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.021) gunicorn-web stdout | 2023-01-17 06:02:37,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:37,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) queuecleanupworker stdout | 2023-01-17 06:02:38,458 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:02:38,462 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 06:02:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:43.935629+00:00 (in 5.200159 seconds) notificationworker stdout | 2023-01-17 06:02:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:48 UTC)" (scheduled at 2023-01-17 06:02:38.735066+00:00) notificationworker stdout | 2023-01-17 06:02:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 38, 735828), True, datetime.datetime(2023, 1, 17, 6, 2, 38, 735828), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:02:38,820 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:02:38,822 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:02:38,829 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:02:38,833 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:02:38,837 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:02:38,840 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:02:38,842 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:02:38,844 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:02:38,848 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:02:38,891 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:02:38,896 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:02:38,900 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:02:38,903 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:02:38,906 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:02:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:02:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:48.738458+00:00 (in 9.707228 seconds) repositorygcworker stdout | 2023-01-17 06:02:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:39 UTC)" (scheduled at 2023-01-17 06:02:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:02:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:02:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 39, 31636), True, datetime.datetime(2023, 1, 17, 6, 2, 39, 31636), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:02:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:02:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:02:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:02:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:52.837365+00:00 (in 13.606891 seconds) securityworker stdout | 2023-01-17 06:02:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:09 UTC)" (scheduled at 2023-01-17 06:02:39.230038+00:00) securityworker stdout | 2023-01-17 06:02:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:02:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:02:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:02:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:02:39,609 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:02:40,244 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:02:41,035 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:02:41,096 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:02:41,117 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:02:41,244 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:02:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:48.735066+00:00 (in 4.798955 seconds) notificationworker stdout | 2023-01-17 06:02:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:43 UTC)" (scheduled at 2023-01-17 06:02:43.935629+00:00) notificationworker stdout | 2023-01-17 06:02:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:02:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:43 UTC)" executed successfully builder stdout | 2023-01-17 06:02:45,065 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:45,075 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:45,075 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:45,081 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:02:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:02:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:59.337413+00:00 (in 11.404499 seconds) namespacegcworker stdout | 2023-01-17 06:02:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:47 UTC)" (scheduled at 2023-01-17 06:02:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:02:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:02:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:58.735066+00:00 (in 9.999452 seconds) notificationworker stdout | 2023-01-17 06:02:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:58 UTC)" (scheduled at 2023-01-17 06:02:48.735066+00:00) notificationworker stdout | 2023-01-17 06:02:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 48, 736046), True, datetime.datetime(2023, 1, 17, 6, 2, 48, 736046), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:02:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:02:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:39.030814+00:00 (in 50.291909 seconds) repositorygcworker stdout | 2023-01-17 06:02:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:48 UTC)" (scheduled at 2023-01-17 06:02:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:02:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:02:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:02:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:02:48,831 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:24.641816+00:00 (in 35.810948 seconds) exportactionlogsworker stdout | 2023-01-17 06:02:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:48 UTC)" (scheduled at 2023-01-17 06:02:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:02:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:02:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 48, 831252), True, datetime.datetime(2023, 1, 17, 6, 2, 48, 831252), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:02:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:02:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:02:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:02:52,624 [243] [DEBUG] [app] Starting request: urn:request:b209ad56-dcf2-4546-86da-33641c130e0f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:52,625 [244] [DEBUG] [app] Starting request: urn:request:65f8e2be-b2cd-42fd-ad50-a78c412b7a6d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:02:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:52,629 [232] [DEBUG] [app] Starting request: urn:request:12e4da87-5c49-43ae-a619-84aead811492 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:52,630 [242] [DEBUG] [app] Starting request: urn:request:70ba9a49-3c42-44d5-945a-de3c89912d3b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:52,630 [232] [DEBUG] [app] Ending request: urn:request:12e4da87-5c49-43ae-a619-84aead811492 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:12e4da87-5c49-43ae-a619-84aead811492', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:02:52,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:02:52,630 [242] [DEBUG] [app] Ending request: urn:request:70ba9a49-3c42-44d5-945a-de3c89912d3b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:70ba9a49-3c42-44d5-945a-de3c89912d3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:52,630 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:02:52,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:52,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,631 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,633 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:52,634 [244] [DEBUG] [app] Starting request: urn:request:f420e957-e9d8-4d40-9145-76aa7291ae27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:52,634 [244] [DEBUG] [app] Ending request: urn:request:f420e957-e9d8-4d40-9145-76aa7291ae27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f420e957-e9d8-4d40-9145-76aa7291ae27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:52,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:52,635 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,635 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:52,636 [241] [DEBUG] [app] Starting request: urn:request:d7459d29-7b5b-464f-99bb-801ab2339645 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:52,636 [241] [DEBUG] [app] Ending request: urn:request:d7459d29-7b5b-464f-99bb-801ab2339645 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d7459d29-7b5b-464f-99bb-801ab2339645', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:52,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:52,646 [243] [DEBUG] [app] Ending request: urn:request:b209ad56-dcf2-4546-86da-33641c130e0f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b209ad56-dcf2-4546-86da-33641c130e0f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:52,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:52,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 06:02:52,647 [244] [DEBUG] [app] Ending request: urn:request:65f8e2be-b2cd-42fd-ad50-a78c412b7a6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65f8e2be-b2cd-42fd-ad50-a78c412b7a6d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:52,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:02:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:02:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:09.230038+00:00 (in 16.392262 seconds) securityworker stdout | 2023-01-17 06:02:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:22 UTC)" (scheduled at 2023-01-17 06:02:52.837365+00:00) securityworker stdout | 2023-01-17 06:02:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:22 UTC)" executed successfully builder stdout | 2023-01-17 06:02:55,091 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:55,100 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:55,100 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:55,106 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:02:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:02:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:27.226424+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 06:02:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:27 UTC)" (scheduled at 2023-01-17 06:02:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:02:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 2, 57, 227286), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:02:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:02:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:02:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:08.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 06:02:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:08 UTC)" (scheduled at 2023-01-17 06:02:58.735066+00:00) notificationworker stdout | 2023-01-17 06:02:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 58, 735955), True, datetime.datetime(2023, 1, 17, 6, 2, 58, 735955), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:02:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:02:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:47.932490+00:00 (in 48.594654 seconds) namespacegcworker stdout | 2023-01-17 06:02:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:59 UTC)" (scheduled at 2023-01-17 06:02:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:02:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:02:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 59, 338206), True, datetime.datetime(2023, 1, 17, 6, 2, 59, 338206), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:02:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:02:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:02:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:03:02,109 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:03:02,309 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:03:02,535 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} globalpromstats stdout | 2023-01-17 06:03:02,827 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 07:03:02.826956+00:00 (in 3599.999070 seconds) globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2023-01-17 07:03:02 UTC)" (scheduled at 2023-01-17 06:03:02.826956+00:00) globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [DEBUG] [__main__] Attempting to report stats globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ... globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 06:03:02,828 [67] [DEBUG] [__main__] Reporting global stats globalpromstats stdout | 2023-01-17 06:03:02,829 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2023-01-17 06:03:02,836 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False]) globalpromstats stdout | 2023-01-17 06:03:02,838 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True]) globalpromstats stdout | 2023-01-17 06:03:02,840 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2023-01-17 06:03:02,841 [67] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:03:02,841 [67] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 06:03:02,841 [67] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2023-01-17 06:03:02,841 [67] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 06:03:02,841 [67] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2023-01-17 07:03:02 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 06:03:03,397 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:03:03,408 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:03:04,657 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:03:05,116 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:05,125 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:05,125 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:05,130 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:03:05,366 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:03:05,411 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:03:05,811 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:03:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:03:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:35.848675+00:00 (in 29.999581 seconds) gcworker stdout | 2023-01-17 06:03:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:35 UTC)" (scheduled at 2023-01-17 06:03:05.848675+00:00) gcworker stdout | 2023-01-17 06:03:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725785849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:03:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:03:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:03:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:03:06,378 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:03:06,381 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:03:06,383 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:03:06,752 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:03:06,838 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:03:07,625 [239] [DEBUG] [app] Starting request: urn:request:4355173f-9342-402b-b005-f71580929e5f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:07,625 [244] [DEBUG] [app] Starting request: urn:request:6cace11e-e26b-464c-a734-591c87224b35 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:07,630 [232] [DEBUG] [app] Starting request: urn:request:6b7b3c9d-9a3d-4e0a-892a-cd9fa265e4a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:07,630 [240] [DEBUG] [app] Starting request: urn:request:cf2426f3-45ef-464f-aa64-859d4b8b5213 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:07,630 [232] [DEBUG] [app] Ending request: urn:request:6b7b3c9d-9a3d-4e0a-892a-cd9fa265e4a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b7b3c9d-9a3d-4e0a-892a-cd9fa265e4a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:03:07,631 [240] [DEBUG] [app] Ending request: urn:request:cf2426f3-45ef-464f-aa64-859d4b8b5213 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf2426f3-45ef-464f-aa64-859d4b8b5213', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:03:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:03:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:07,636 [244] [DEBUG] [app] Starting request: urn:request:ceee4b9a-ead3-47cd-8238-4f5703453e34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:07,635 [243] [DEBUG] [app] Starting request: urn:request:7e7d6341-2488-4665-8410-43332b29f11e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:07,636 [244] [DEBUG] [app] Ending request: urn:request:ceee4b9a-ead3-47cd-8238-4f5703453e34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ceee4b9a-ead3-47cd-8238-4f5703453e34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:07,636 [243] [DEBUG] [app] Ending request: urn:request:7e7d6341-2488-4665-8410-43332b29f11e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e7d6341-2488-4665-8410-43332b29f11e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:03:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:07,647 [239] [DEBUG] [app] Ending request: urn:request:4355173f-9342-402b-b005-f71580929e5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4355173f-9342-402b-b005-f71580929e5f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:07,647 [244] [DEBUG] [app] Ending request: urn:request:6cace11e-e26b-464c-a734-591c87224b35 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cace11e-e26b-464c-a734-591c87224b35', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:03:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:03:07,655 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:03:08,470 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:03:08,472 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:03:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:18.735066+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 06:03:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:18 UTC)" (scheduled at 2023-01-17 06:03:08.735066+00:00) notificationworker stdout | 2023-01-17 06:03:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 8, 735942), True, datetime.datetime(2023, 1, 17, 6, 3, 8, 735942), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:03:08,826 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:03:08,830 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:03:08,838 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:03:08,841 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:03:08,844 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:03:08,848 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:03:08,850 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:03:08,853 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:03:08,856 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:03:08,897 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:03:08,902 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:03:08,906 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:03:08,909 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:03:08,913 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:03:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:22.837365+00:00 (in 13.606864 seconds) securityworker stdout | 2023-01-17 06:03:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:39 UTC)" (scheduled at 2023-01-17 06:03:09.230038+00:00) securityworker stdout | 2023-01-17 06:03:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:03:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:03:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:03:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:03:09,628 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:03:10,256 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:03:11,070 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:03:11,126 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 06:03:11,130 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 06:03:11,269 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:03:15,141 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:15,150 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:15,150 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:15,156 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:28.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 06:03:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:28 UTC)" (scheduled at 2023-01-17 06:03:18.735066+00:00) notificationworker stdout | 2023-01-17 06:03:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 18, 735952), True, datetime.datetime(2023, 1, 17, 6, 3, 18, 735952), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:03:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:03:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:25.334138+00:00 (in 5.091484 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:03:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:20 UTC)" (scheduled at 2023-01-17 06:03:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:03:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:03:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 20, 243119), True, datetime.datetime(2023, 1, 17, 6, 3, 20, 243119), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:03:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:03:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:03:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:03:22,625 [244] [DEBUG] [app] Starting request: urn:request:8a097b0e-85c2-45bc-9b12-8c04e9b16ca0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:22,625 [243] [DEBUG] [app] Starting request: urn:request:584abf54-88dd-416e-94b9-ea9c68832ae4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:22,630 [232] [DEBUG] [app] Starting request: urn:request:34aa8d54-d547-4761-bc41-e719ac97c860 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:22,630 [240] [DEBUG] [app] Starting request: urn:request:48d6d5f5-1ed9-400e-952a-2813088dc527 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:22,631 [232] [DEBUG] [app] Ending request: urn:request:34aa8d54-d547-4761-bc41-e719ac97c860 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34aa8d54-d547-4761-bc41-e719ac97c860', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:03:22,631 [240] [DEBUG] [app] Ending request: urn:request:48d6d5f5-1ed9-400e-952a-2813088dc527 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48d6d5f5-1ed9-400e-952a-2813088dc527', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:03:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:03:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:03:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:22,636 [239] [DEBUG] [app] Starting request: urn:request:a8b5d135-34df-4216-b88b-aa932ca33187 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:22,636 [241] [DEBUG] [app] Starting request: urn:request:59c89951-bee7-4326-a076-7f2eb7c68834 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:22,636 [239] [DEBUG] [app] Ending request: urn:request:a8b5d135-34df-4216-b88b-aa932ca33187 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a8b5d135-34df-4216-b88b-aa932ca33187', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:22,636 [241] [DEBUG] [app] Ending request: urn:request:59c89951-bee7-4326-a076-7f2eb7c68834 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59c89951-bee7-4326-a076-7f2eb7c68834', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:03:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:22,648 [244] [DEBUG] [app] Ending request: urn:request:8a097b0e-85c2-45bc-9b12-8c04e9b16ca0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a097b0e-85c2-45bc-9b12-8c04e9b16ca0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:22,648 [243] [DEBUG] [app] Ending request: urn:request:584abf54-88dd-416e-94b9-ea9c68832ae4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:584abf54-88dd-416e-94b9-ea9c68832ae4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:03:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:03:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:39.230038+00:00 (in 16.392203 seconds) securityworker stdout | 2023-01-17 06:03:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:52 UTC)" (scheduled at 2023-01-17 06:03:22.837365+00:00) securityworker stdout | 2023-01-17 06:03:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:03:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:03:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:48.830274+00:00 (in 24.188028 seconds) exportactionlogsworker stdout | 2023-01-17 06:03:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:24 UTC)" (scheduled at 2023-01-17 06:03:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:03:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:03:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:24 UTC)" executed successfully builder stdout | 2023-01-17 06:03:25,166 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:25,175 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:25,175 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:25,181 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:03:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:03:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:20.242198+00:00 (in 54.907492 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:03:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:25 UTC)" (scheduled at 2023-01-17 06:03:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:03:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:03:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:03:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:03:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:57.226424+00:00 (in 29.999554 seconds) buildlogsarchiver stdout | 2023-01-17 06:03:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:57 UTC)" (scheduled at 2023-01-17 06:03:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:03:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 3, 27, 227221), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:03:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:03:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:03:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:38.735066+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 06:03:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:38 UTC)" (scheduled at 2023-01-17 06:03:28.735066+00:00) notificationworker stdout | 2023-01-17 06:03:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 28, 735993), True, datetime.datetime(2023, 1, 17, 6, 3, 28, 735993), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:03:32,134 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:03:32,343 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:03:32,565 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} teamsyncworker stdout | 2023-01-17 06:03:33,429 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} buildlogsarchiver stdout | 2023-01-17 06:03:33,434 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 06:03:34,693 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:03:35,191 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:35,209 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:35,209 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:35,215 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:03:35,380 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:03:35,446 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:03:35,838 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:03:35,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:03:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:05.848675+00:00 (in 29.999577 seconds) gcworker stdout | 2023-01-17 06:03:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:05 UTC)" (scheduled at 2023-01-17 06:03:35.848675+00:00) gcworker stdout | 2023-01-17 06:03:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725815849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:03:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:03:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:03:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:05 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:03:36,386 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:03:36,388 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:03:36,390 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:03:36,774 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:03:36,873 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:03:37,625 [241] [DEBUG] [app] Starting request: urn:request:97f66b8d-4000-4918-9a5d-f4c07c1d8758 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:37,625 [243] [DEBUG] [app] Starting request: urn:request:10c69984-a937-438b-b629-59ff598e1169 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:37,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:37,630 [237] [DEBUG] [app] Starting request: urn:request:81bd48d2-b828-4871-965d-29a1d7833241 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:37,630 [232] [DEBUG] [app] Starting request: urn:request:d8c902a5-7670-4d08-b476-1b46f022b885 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:37,630 [237] [DEBUG] [app] Ending request: urn:request:81bd48d2-b828-4871-965d-29a1d7833241 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81bd48d2-b828-4871-965d-29a1d7833241', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:37,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:03:37,631 [232] [DEBUG] [app] Ending request: urn:request:d8c902a5-7670-4d08-b476-1b46f022b885 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8c902a5-7670-4d08-b476-1b46f022b885', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:03:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:03:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:03:37,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:37,635 [244] [DEBUG] [app] Starting request: urn:request:69429779-4917-4917-9511-945c4486ce81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:37,635 [239] [DEBUG] [app] Starting request: urn:request:ce3265d1-f64a-4157-be27-a3f0e92c4e9d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:37,635 [244] [DEBUG] [app] Ending request: urn:request:69429779-4917-4917-9511-945c4486ce81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:69429779-4917-4917-9511-945c4486ce81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:37,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:37,635 [239] [DEBUG] [app] Ending request: urn:request:ce3265d1-f64a-4157-be27-a3f0e92c4e9d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ce3265d1-f64a-4157-be27-a3f0e92c4e9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:37,635 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:37,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:37,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:37,640 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:37,646 [243] [DEBUG] [app] Ending request: urn:request:10c69984-a937-438b-b629-59ff598e1169 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10c69984-a937-438b-b629-59ff598e1169', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:37,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:37,646 [241] [DEBUG] [app] Ending request: urn:request:97f66b8d-4000-4918-9a5d-f4c07c1d8758 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97f66b8d-4000-4918-9a5d-f4c07c1d8758', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:37,646 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:37,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 06:03:37,646 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:03:37,690 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:03:38,476 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:03:38,507 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:03:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:43.935629+00:00 (in 5.200116 seconds) notificationworker stdout | 2023-01-17 06:03:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:48 UTC)" (scheduled at 2023-01-17 06:03:38.735066+00:00) notificationworker stdout | 2023-01-17 06:03:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 38, 735899), True, datetime.datetime(2023, 1, 17, 6, 3, 38, 735899), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:03:38,838 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:03:38,841 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:03:38,845 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:03:38,849 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:03:38,854 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:03:38,856 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:03:38,858 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:03:38,861 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:03:38,864 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:03:38,903 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:03:38,907 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:03:38,912 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:03:38,916 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:03:38,919 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:03:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:03:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:48.738458+00:00 (in 9.707213 seconds) repositorygcworker stdout | 2023-01-17 06:03:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:39 UTC)" (scheduled at 2023-01-17 06:03:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:03:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:03:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 39, 31682), True, datetime.datetime(2023, 1, 17, 6, 3, 39, 31682), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:03:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:03:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:03:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:03:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:52.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 06:03:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:09 UTC)" (scheduled at 2023-01-17 06:03:39.230038+00:00) securityworker stdout | 2023-01-17 06:03:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:03:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:03:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:39,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:39,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:03:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:03:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:03:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:03:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:03:39,663 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:03:40,268 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:03:41,106 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:03:41,136 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} expiredappspecifictokenworker stdout | 2023-01-17 06:03:41,144 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} securityscanningnotificationworker stdout | 2023-01-17 06:03:41,295 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:03:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:48.735066+00:00 (in 4.798958 seconds) notificationworker stdout | 2023-01-17 06:03:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:43 UTC)" (scheduled at 2023-01-17 06:03:43.935629+00:00) notificationworker stdout | 2023-01-17 06:03:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:03:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:43 UTC)" executed successfully builder stdout | 2023-01-17 06:03:45,225 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:45,234 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:45,234 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:45,239 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:03:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:03:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:59.337413+00:00 (in 11.404479 seconds) namespacegcworker stdout | 2023-01-17 06:03:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:47 UTC)" (scheduled at 2023-01-17 06:03:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:03:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:03:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:56.935055+00:00 (in 8.199540 seconds) notificationworker stdout | 2023-01-17 06:03:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:58 UTC)" (scheduled at 2023-01-17 06:03:48.735066+00:00) notificationworker stdout | 2023-01-17 06:03:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 48, 735878), True, datetime.datetime(2023, 1, 17, 6, 3, 48, 735878), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:03:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:03:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:39.030814+00:00 (in 50.291848 seconds) repositorygcworker stdout | 2023-01-17 06:03:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:48 UTC)" (scheduled at 2023-01-17 06:03:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:03:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:03:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:03:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:03:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:24.641816+00:00 (in 35.811074 seconds) exportactionlogsworker stdout | 2023-01-17 06:03:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:48 UTC)" (scheduled at 2023-01-17 06:03:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:03:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:03:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 48, 831115), True, datetime.datetime(2023, 1, 17, 6, 3, 48, 831115), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:03:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:03:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:03:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:03:52,625 [239] [DEBUG] [app] Starting request: urn:request:400f1321-cba0-45da-885f-15eb02e017cb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:52,625 [241] [DEBUG] [app] Starting request: urn:request:53e33d58-c643-48b9-ae3a-230e7453451d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:03:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:52,630 [240] [DEBUG] [app] Starting request: urn:request:2b11af2f-60d1-479c-9719-401a00c54906 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:52,630 [232] [DEBUG] [app] Starting request: urn:request:b6ecabb7-9e29-483e-a273-8d84b7e7ad65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:52,631 [240] [DEBUG] [app] Ending request: urn:request:2b11af2f-60d1-479c-9719-401a00c54906 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2b11af2f-60d1-479c-9719-401a00c54906', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:03:52,631 [232] [DEBUG] [app] Ending request: urn:request:b6ecabb7-9e29-483e-a273-8d84b7e7ad65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6ecabb7-9e29-483e-a273-8d84b7e7ad65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:03:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:52,636 [244] [DEBUG] [app] Starting request: urn:request:95938de5-33f6-4196-881a-bed5a8ea1b3b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:52,636 [243] [DEBUG] [app] Starting request: urn:request:bef6b774-d605-4123-891c-354a69c06487 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:52,636 [244] [DEBUG] [app] Ending request: urn:request:95938de5-33f6-4196-881a-bed5a8ea1b3b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95938de5-33f6-4196-881a-bed5a8ea1b3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:52,636 [243] [DEBUG] [app] Ending request: urn:request:bef6b774-d605-4123-891c-354a69c06487 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bef6b774-d605-4123-891c-354a69c06487', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:52,648 [241] [DEBUG] [app] Ending request: urn:request:53e33d58-c643-48b9-ae3a-230e7453451d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:53e33d58-c643-48b9-ae3a-230e7453451d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:52,648 [239] [DEBUG] [app] Ending request: urn:request:400f1321-cba0-45da-885f-15eb02e017cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:400f1321-cba0-45da-885f-15eb02e017cb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:03:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:03:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:09.230038+00:00 (in 16.392243 seconds) securityworker stdout | 2023-01-17 06:03:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:22 UTC)" (scheduled at 2023-01-17 06:03:52.837365+00:00) securityworker stdout | 2023-01-17 06:03:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:22 UTC)" executed successfully builder stdout | 2023-01-17 06:03:55,249 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:55,258 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:55,259 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:55,264 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:58.735066+00:00 (in 1.799584 seconds) notificationworker stdout | 2023-01-17 06:03:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:08:56 UTC)" (scheduled at 2023-01-17 06:03:56.935055+00:00) notificationworker stdout | 2023-01-17 06:03:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 3, 56, 935829), 'notification/%']) notificationworker stdout | 2023-01-17 06:03:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 3, 56, 935829), True, datetime.datetime(2023, 1, 17, 6, 3, 56, 935829), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:03:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 3, 56, 935829), True, datetime.datetime(2023, 1, 17, 6, 3, 56, 935829), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 3, 56, 935829), 'notification/%']) notificationworker stdout | 2023-01-17 06:03:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:08:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:03:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:03:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:27.226424+00:00 (in 29.999570 seconds) buildlogsarchiver stdout | 2023-01-17 06:03:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:27 UTC)" (scheduled at 2023-01-17 06:03:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:03:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 3, 57, 227221), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:03:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:03:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:03:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:08.735066+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 06:03:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:08 UTC)" (scheduled at 2023-01-17 06:03:58.735066+00:00) notificationworker stdout | 2023-01-17 06:03:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 58, 735946), True, datetime.datetime(2023, 1, 17, 6, 3, 58, 735946), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:03:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:03:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:15.931994+00:00 (in 16.594103 seconds) namespacegcworker stdout | 2023-01-17 06:03:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:59 UTC)" (scheduled at 2023-01-17 06:03:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:03:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:03:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 59, 338328), True, datetime.datetime(2023, 1, 17, 6, 3, 59, 338328), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:03:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:03:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:03:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:04:02,144 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:04:02,359 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:04:02,579 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:04:03,451 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:04:03,463 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:04:04,718 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:04:05,266 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:05,274 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:05,274 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:05,280 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:04:05,416 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:04:05,480 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:04:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:04:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:35.848675+00:00 (in 29.999587 seconds) gcworker stdout | 2023-01-17 06:04:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:35 UTC)" (scheduled at 2023-01-17 06:04:05.848675+00:00) gcworker stdout | 2023-01-17 06:04:05,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) namespacegcworker stdout | 2023-01-17 06:04:05,852 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gcworker stdout | 2023-01-17 06:04:05,856 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725845855, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:04:05,859 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:04:05,859 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:04:05,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:35 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:04:06,392 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:04:06,395 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:04:06,397 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:04:06,790 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:04:06,877 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:04:07,625 [239] [DEBUG] [app] Starting request: urn:request:a2b5830d-6b6f-44c3-b1a9-953b2eb4c35a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:07,625 [241] [DEBUG] [app] Starting request: urn:request:1ca4a393-8389-47ec-8c07-a5d28faf9d23 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:07,630 [232] [DEBUG] [app] Starting request: urn:request:b3983456-3418-425e-bec7-9e31fabfbe62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,630 [240] [DEBUG] [app] Starting request: urn:request:48dfbea1-be80-46eb-8d8c-3e42f9dbd566 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,630 [232] [DEBUG] [app] Ending request: urn:request:b3983456-3418-425e-bec7-9e31fabfbe62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3983456-3418-425e-bec7-9e31fabfbe62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,630 [240] [DEBUG] [app] Ending request: urn:request:48dfbea1-be80-46eb-8d8c-3e42f9dbd566 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48dfbea1-be80-46eb-8d8c-3e42f9dbd566', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:04:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:04:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:07,635 [239] [DEBUG] [app] Starting request: urn:request:6ab07b1a-1185-48ce-ba4a-4888dbd4a890 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:07,635 [244] [DEBUG] [app] Starting request: urn:request:aa96410a-075e-4ec8-8751-63ec4607c8af (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:07,636 [239] [DEBUG] [app] Ending request: urn:request:6ab07b1a-1185-48ce-ba4a-4888dbd4a890 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ab07b1a-1185-48ce-ba4a-4888dbd4a890', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:07,636 [244] [DEBUG] [app] Ending request: urn:request:aa96410a-075e-4ec8-8751-63ec4607c8af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aa96410a-075e-4ec8-8751-63ec4607c8af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:04:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:07,646 [239] [DEBUG] [app] Ending request: urn:request:a2b5830d-6b6f-44c3-b1a9-953b2eb4c35a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2b5830d-6b6f-44c3-b1a9-953b2eb4c35a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:07,647 [241] [DEBUG] [app] Ending request: urn:request:1ca4a393-8389-47ec-8c07-a5d28faf9d23 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ca4a393-8389-47ec-8c07-a5d28faf9d23', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:04:07,647 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 06:04:07,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:04:07,718 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:04:08,512 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:04:08,537 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:04:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:18.735066+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 06:04:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:18 UTC)" (scheduled at 2023-01-17 06:04:08.735066+00:00) notificationworker stdout | 2023-01-17 06:04:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 8, 735935), True, datetime.datetime(2023, 1, 17, 6, 4, 8, 735935), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:04:08,848 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:04:08,850 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:04:08,853 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:04:08,856 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:04:08,861 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:04:08,864 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:04:08,867 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:04:08,869 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:04:08,871 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:04:08,909 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:04:08,913 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:04:08,917 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:04:08,922 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:04:08,926 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:04:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:22.837365+00:00 (in 13.606885 seconds) securityworker stdout | 2023-01-17 06:04:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:39 UTC)" (scheduled at 2023-01-17 06:04:09.230038+00:00) securityworker stdout | 2023-01-17 06:04:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:04:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:04:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:04:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:04:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:04:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:04:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:04:09,674 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:04:10,296 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:04:11,111 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:04:11,153 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:04:11,171 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:04:11,309 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:04:15,290 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:15,300 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:15,300 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:15,305 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:04:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:04:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:47.932490+00:00 (in 31.999990 seconds) namespacegcworker stdout | 2023-01-17 06:04:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:09:15 UTC)" (scheduled at 2023-01-17 06:04:15.931994+00:00) namespacegcworker stdout | 2023-01-17 06:04:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 4, 15, 932890), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:04:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 4, 15, 932890), True, datetime.datetime(2023, 1, 17, 6, 4, 15, 932890), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:04:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 4, 15, 932890), True, datetime.datetime(2023, 1, 17, 6, 4, 15, 932890), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 4, 15, 932890), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:04:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:04:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:09:15 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:28.735066+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 06:04:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:28 UTC)" (scheduled at 2023-01-17 06:04:18.735066+00:00) notificationworker stdout | 2023-01-17 06:04:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 18, 735975), True, datetime.datetime(2023, 1, 17, 6, 4, 18, 735975), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:04:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:04:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:25.334138+00:00 (in 5.091445 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:04:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:20 UTC)" (scheduled at 2023-01-17 06:04:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:04:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:04:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 20, 243124), True, datetime.datetime(2023, 1, 17, 6, 4, 20, 243124), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:04:20,252 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:04:20,252 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:04:20,252 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:04:22,625 [244] [DEBUG] [app] Starting request: urn:request:3ceddb09-12c1-48e1-9ab6-70a2f323d781 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:22,625 [239] [DEBUG] [app] Starting request: urn:request:a700a785-2d71-475b-b6d2-77c85e5a5ac6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:22,630 [240] [DEBUG] [app] Starting request: urn:request:2c494d2a-a784-4806-9661-c4a08969050c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:22,631 [236] [DEBUG] [app] Starting request: urn:request:6fbf3796-d4dc-443d-a207-44a0523a4ca2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:22,631 [240] [DEBUG] [app] Ending request: urn:request:2c494d2a-a784-4806-9661-c4a08969050c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c494d2a-a784-4806-9661-c4a08969050c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:04:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:04:22,631 [236] [DEBUG] [app] Ending request: urn:request:6fbf3796-d4dc-443d-a207-44a0523a4ca2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6fbf3796-d4dc-443d-a207-44a0523a4ca2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:04:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:22,635 [243] [DEBUG] [app] Starting request: urn:request:2f921e13-082d-4b1c-b4e9-e439aff2822c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:22,636 [244] [DEBUG] [app] Starting request: urn:request:7ae80d20-7fe1-4c6c-ba2a-f62a8b4acfbe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:22,636 [243] [DEBUG] [app] Ending request: urn:request:2f921e13-082d-4b1c-b4e9-e439aff2822c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f921e13-082d-4b1c-b4e9-e439aff2822c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:22,636 [244] [DEBUG] [app] Ending request: urn:request:7ae80d20-7fe1-4c6c-ba2a-f62a8b4acfbe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ae80d20-7fe1-4c6c-ba2a-f62a8b4acfbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:04:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:22,648 [244] [DEBUG] [app] Ending request: urn:request:3ceddb09-12c1-48e1-9ab6-70a2f323d781 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ceddb09-12c1-48e1-9ab6-70a2f323d781', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:22,648 [239] [DEBUG] [app] Ending request: urn:request:a700a785-2d71-475b-b6d2-77c85e5a5ac6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a700a785-2d71-475b-b6d2-77c85e5a5ac6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:04:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:04:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:39.230038+00:00 (in 16.392232 seconds) securityworker stdout | 2023-01-17 06:04:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:52 UTC)" (scheduled at 2023-01-17 06:04:22.837365+00:00) securityworker stdout | 2023-01-17 06:04:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:04:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:04:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:48.830274+00:00 (in 24.187996 seconds) exportactionlogsworker stdout | 2023-01-17 06:04:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:24 UTC)" (scheduled at 2023-01-17 06:04:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:04:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:04:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:24 UTC)" executed successfully builder stdout | 2023-01-17 06:04:25,316 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:25,325 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:25,325 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:25,331 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:04:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:04:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:20.242198+00:00 (in 54.907622 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:04:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:25 UTC)" (scheduled at 2023-01-17 06:04:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:04:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:04:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:04:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:04:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:57.226424+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 06:04:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:57 UTC)" (scheduled at 2023-01-17 06:04:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:04:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 4, 27, 227237), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:04:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:04:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:04:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:38.735066+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 06:04:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:38 UTC)" (scheduled at 2023-01-17 06:04:28.735066+00:00) notificationworker stdout | 2023-01-17 06:04:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 28, 735922), True, datetime.datetime(2023, 1, 17, 6, 4, 28, 735922), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:04:32,180 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:04:32,394 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:04:32,614 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:04:33,486 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:04:33,491 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:04:34,731 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:04:35,341 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:35,350 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:35,350 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:35,356 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:04:35,452 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:04:35,515 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:04:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:04:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:05.848675+00:00 (in 29.999496 seconds) gcworker stdout | 2023-01-17 06:04:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:05 UTC)" (scheduled at 2023-01-17 06:04:35.848675+00:00) gcworker stdout | 2023-01-17 06:04:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725875849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:04:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:04:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:04:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:05 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:04:35,887 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:04:36,399 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:04:36,402 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:04:36,404 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:04:36,817 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:04:36,912 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:04:37,625 [239] [DEBUG] [app] Starting request: urn:request:f57bb343-3795-4f8a-9fb0-c25ece132941 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:37,625 [244] [DEBUG] [app] Starting request: urn:request:b13065ae-9111-452e-92f2-060b37a887f5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:37,630 [240] [DEBUG] [app] Starting request: urn:request:40187460-0b36-4cfb-9b40-aa1c7aeb17ee (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:37,630 [232] [DEBUG] [app] Starting request: urn:request:9eb15ab5-1116-4f6e-97db-0c539e8756c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:37,630 [240] [DEBUG] [app] Ending request: urn:request:40187460-0b36-4cfb-9b40-aa1c7aeb17ee (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40187460-0b36-4cfb-9b40-aa1c7aeb17ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:37,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:04:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:37,631 [232] [DEBUG] [app] Ending request: urn:request:9eb15ab5-1116-4f6e-97db-0c539e8756c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9eb15ab5-1116-4f6e-97db-0c539e8756c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:04:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:37,635 [243] [DEBUG] [app] Starting request: urn:request:2362bb2e-4319-4f8e-8725-a125a250e65a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:37,635 [241] [DEBUG] [app] Starting request: urn:request:bdaf58b0-e537-435c-b890-b227b8bbf306 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:37,635 [243] [DEBUG] [app] Ending request: urn:request:2362bb2e-4319-4f8e-8725-a125a250e65a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2362bb2e-4319-4f8e-8725-a125a250e65a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:37,636 [241] [DEBUG] [app] Ending request: urn:request:bdaf58b0-e537-435c-b890-b227b8bbf306 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bdaf58b0-e537-435c-b890-b227b8bbf306', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:04:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:37,647 [244] [DEBUG] [app] Ending request: urn:request:b13065ae-9111-452e-92f2-060b37a887f5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b13065ae-9111-452e-92f2-060b37a887f5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:37,647 [239] [DEBUG] [app] Ending request: urn:request:f57bb343-3795-4f8a-9fb0-c25ece132941 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f57bb343-3795-4f8a-9fb0-c25ece132941', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:04:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:04:37,727 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:04:38,518 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:04:38,565 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:04:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:43.935629+00:00 (in 5.200107 seconds) notificationworker stdout | 2023-01-17 06:04:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:48 UTC)" (scheduled at 2023-01-17 06:04:38.735066+00:00) notificationworker stdout | 2023-01-17 06:04:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 38, 735877), True, datetime.datetime(2023, 1, 17, 6, 4, 38, 735877), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:04:38,858 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:04:38,860 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:04:38,862 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:04:38,865 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:04:38,868 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:04:38,871 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:04:38,876 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:04:38,879 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:04:38,881 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:04:38,915 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:04:38,919 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:04:38,923 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:04:38,928 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:04:38,932 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:04:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:04:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:48.738458+00:00 (in 9.707022 seconds) repositorygcworker stdout | 2023-01-17 06:04:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:39 UTC)" (scheduled at 2023-01-17 06:04:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:04:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:04:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 39, 31878), True, datetime.datetime(2023, 1, 17, 6, 4, 39, 31878), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:04:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:04:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:04:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:04:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:52.837365+00:00 (in 13.606887 seconds) securityworker stdout | 2023-01-17 06:04:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:09 UTC)" (scheduled at 2023-01-17 06:04:39.230038+00:00) securityworker stdout | 2023-01-17 06:04:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:04:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:04:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:39,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:39,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:04:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:04:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:04:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:04:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:04:39,681 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:04:40,318 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:04:41,147 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:04:41,160 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:04:41,205 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:04:41,337 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:04:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:48.735066+00:00 (in 4.798980 seconds) notificationworker stdout | 2023-01-17 06:04:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:43 UTC)" (scheduled at 2023-01-17 06:04:43.935629+00:00) notificationworker stdout | 2023-01-17 06:04:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:04:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:43 UTC)" executed successfully builder stdout | 2023-01-17 06:04:45,366 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:45,375 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:45,375 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:45,381 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:04:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:04:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:59.337413+00:00 (in 11.404433 seconds) namespacegcworker stdout | 2023-01-17 06:04:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:47 UTC)" (scheduled at 2023-01-17 06:04:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:04:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:04:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:58.735066+00:00 (in 9.999490 seconds) notificationworker stdout | 2023-01-17 06:04:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:58 UTC)" (scheduled at 2023-01-17 06:04:48.735066+00:00) notificationworker stdout | 2023-01-17 06:04:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 48, 735983), True, datetime.datetime(2023, 1, 17, 6, 4, 48, 735983), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:04:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:04:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:39.030814+00:00 (in 50.291846 seconds) repositorygcworker stdout | 2023-01-17 06:04:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:48 UTC)" (scheduled at 2023-01-17 06:04:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:04:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:04:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:04:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:04:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:24.641816+00:00 (in 35.811058 seconds) exportactionlogsworker stdout | 2023-01-17 06:04:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:48 UTC)" (scheduled at 2023-01-17 06:04:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:04:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:04:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 48, 831109), True, datetime.datetime(2023, 1, 17, 6, 4, 48, 831109), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:04:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:04:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:04:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:04:52,626 [239] [DEBUG] [app] Starting request: urn:request:b6338879-4d7c-4445-b665-65a28c5733dc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:52,626 [243] [DEBUG] [app] Starting request: urn:request:c0170dfe-9b7a-41fd-b566-236a822b7fdb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:04:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,628 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:52,630 [240] [DEBUG] [app] Starting request: urn:request:51884c24-f18a-4b45-b485-09fbbbe795c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,631 [240] [DEBUG] [app] Ending request: urn:request:51884c24-f18a-4b45-b485-09fbbbe795c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51884c24-f18a-4b45-b485-09fbbbe795c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,631 [232] [DEBUG] [app] Starting request: urn:request:ea522e16-3086-42d2-a0e3-284340ae3a8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:04:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:52,632 [232] [DEBUG] [app] Ending request: urn:request:ea522e16-3086-42d2-a0e3-284340ae3a8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea522e16-3086-42d2-a0e3-284340ae3a8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:04:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:52,637 [244] [DEBUG] [app] Starting request: urn:request:00f03616-7cdf-44b8-afd7-b592124d59a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:52,637 [244] [DEBUG] [app] Ending request: urn:request:00f03616-7cdf-44b8-afd7-b592124d59a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00f03616-7cdf-44b8-afd7-b592124d59a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:52,637 [241] [DEBUG] [app] Starting request: urn:request:8be19ff4-8c6b-42b3-ba31-f998cfab57e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:52,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:52,638 [241] [DEBUG] [app] Ending request: urn:request:8be19ff4-8c6b-42b3-ba31-f998cfab57e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8be19ff4-8c6b-42b3-ba31-f998cfab57e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:52,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:04:52,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:52,648 [239] [DEBUG] [app] Ending request: urn:request:b6338879-4d7c-4445-b665-65a28c5733dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6338879-4d7c-4445-b665-65a28c5733dc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:52,649 [243] [DEBUG] [app] Ending request: urn:request:c0170dfe-9b7a-41fd-b566-236a822b7fdb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0170dfe-9b7a-41fd-b566-236a822b7fdb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:52,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:04:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:04:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:09.230038+00:00 (in 16.392249 seconds) securityworker stdout | 2023-01-17 06:04:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:22 UTC)" (scheduled at 2023-01-17 06:04:52.837365+00:00) securityworker stdout | 2023-01-17 06:04:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:52,842 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:22 UTC)" executed successfully builder stdout | 2023-01-17 06:04:55,383 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:55,391 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:55,392 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:55,397 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:04:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:04:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:27.226424+00:00 (in 29.999561 seconds) buildlogsarchiver stdout | 2023-01-17 06:04:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:27 UTC)" (scheduled at 2023-01-17 06:04:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:04:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 4, 57, 227190), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:04:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:04:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:04:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:08.735066+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 06:04:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:08 UTC)" (scheduled at 2023-01-17 06:04:58.735066+00:00) notificationworker stdout | 2023-01-17 06:04:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 58, 735885), True, datetime.datetime(2023, 1, 17, 6, 4, 58, 735885), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:04:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:04:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:47.932490+00:00 (in 48.594615 seconds) namespacegcworker stdout | 2023-01-17 06:04:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:59 UTC)" (scheduled at 2023-01-17 06:04:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:04:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:04:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 59, 338261), True, datetime.datetime(2023, 1, 17, 6, 4, 59, 338261), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:04:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:04:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:04:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:05:02,186 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:05:02,410 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:05:02,648 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:05:03,516 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:05:03,526 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:05:04,766 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:05:05,401 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:05,410 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:05,410 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:05,416 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:05:05,488 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:05:05,550 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:05:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:05:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:35.848675+00:00 (in 29.999475 seconds) gcworker stdout | 2023-01-17 06:05:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:35 UTC)" (scheduled at 2023-01-17 06:05:05.848675+00:00) gcworker stdout | 2023-01-17 06:05:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725905849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:05:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:05:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:05:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:35 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:05:05,920 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:05:06,407 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:05:06,409 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:05:06,412 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:05:06,837 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:05:06,933 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:05:07,625 [244] [DEBUG] [app] Starting request: urn:request:6cca9d25-daa6-48d3-a0a6-37440efa1957 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:07,625 [239] [DEBUG] [app] Starting request: urn:request:d37db16b-b063-45eb-a738-8468e3aebe6b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:07,630 [232] [DEBUG] [app] Starting request: urn:request:caec220c-f5fa-4c87-ba79-9d2e58e7a79e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:07,631 [235] [DEBUG] [app] Starting request: urn:request:0d5de93d-a237-4285-8123-422d6120d28b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:07,631 [232] [DEBUG] [app] Ending request: urn:request:caec220c-f5fa-4c87-ba79-9d2e58e7a79e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:caec220c-f5fa-4c87-ba79-9d2e58e7a79e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:05:07,631 [235] [DEBUG] [app] Ending request: urn:request:0d5de93d-a237-4285-8123-422d6120d28b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d5de93d-a237-4285-8123-422d6120d28b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:05:07,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:05:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:07,636 [239] [DEBUG] [app] Starting request: urn:request:d8aeb80f-8760-47b2-a596-866f3ec6fada (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:07,636 [239] [DEBUG] [app] Ending request: urn:request:d8aeb80f-8760-47b2-a596-866f3ec6fada (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8aeb80f-8760-47b2-a596-866f3ec6fada', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:07,636 [241] [DEBUG] [app] Starting request: urn:request:a7d76f39-7d21-493f-adba-2015971f6a16 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:07,637 [241] [DEBUG] [app] Ending request: urn:request:a7d76f39-7d21-493f-adba-2015971f6a16 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7d76f39-7d21-493f-adba-2015971f6a16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:05:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:07,647 [239] [DEBUG] [app] Ending request: urn:request:d37db16b-b063-45eb-a738-8468e3aebe6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d37db16b-b063-45eb-a738-8468e3aebe6b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:07,647 [244] [DEBUG] [app] Ending request: urn:request:6cca9d25-daa6-48d3-a0a6-37440efa1957 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cca9d25-daa6-48d3-a0a6-37440efa1957', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 06:05:07,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:05:07,734 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:05:08,554 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:05:08,599 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:05:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:18.735066+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 06:05:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:18 UTC)" (scheduled at 2023-01-17 06:05:08.735066+00:00) notificationworker stdout | 2023-01-17 06:05:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 8, 735883), True, datetime.datetime(2023, 1, 17, 6, 5, 8, 735883), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:05:08,866 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:05:08,869 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:05:08,872 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:05:08,874 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:05:08,876 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:05:08,879 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:05:08,882 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:05:08,888 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:05:08,891 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:05:08,920 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:05:08,925 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:05:08,929 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:05:08,934 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:05:08,938 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:05:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:22.837365+00:00 (in 13.606890 seconds) securityworker stdout | 2023-01-17 06:05:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:39 UTC)" (scheduled at 2023-01-17 06:05:09.230038+00:00) securityworker stdout | 2023-01-17 06:05:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:05:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:05:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:05:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:05:09,717 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:05:10,327 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:05:11,153 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:05:11,195 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:05:11,240 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:05:11,348 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:05:15,418 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:15,428 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:15,428 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:15,433 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:28.735066+00:00 (in 9.999570 seconds) notificationworker stdout | 2023-01-17 06:05:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:28 UTC)" (scheduled at 2023-01-17 06:05:18.735066+00:00) notificationworker stdout | 2023-01-17 06:05:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 18, 735852), True, datetime.datetime(2023, 1, 17, 6, 5, 18, 735852), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:05:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:05:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:25.334138+00:00 (in 5.091525 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:05:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:20 UTC)" (scheduled at 2023-01-17 06:05:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:05:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:05:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 20, 242983), True, datetime.datetime(2023, 1, 17, 6, 5, 20, 242983), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:05:20,254 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:05:20,254 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:05:20,254 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:05:22,625 [241] [DEBUG] [app] Starting request: urn:request:3446addd-4fc8-4897-9c0e-4cd093a53720 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:22,625 [244] [DEBUG] [app] Starting request: urn:request:cadced52-83dc-4236-85b4-e8b02a80e597 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:22,630 [232] [DEBUG] [app] Starting request: urn:request:0db3c100-3111-4bdc-91ef-09b83a4a124c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:22,630 [235] [DEBUG] [app] Starting request: urn:request:6b9af36d-d503-42f9-84cd-ce3420697f0a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:22,630 [232] [DEBUG] [app] Ending request: urn:request:0db3c100-3111-4bdc-91ef-09b83a4a124c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0db3c100-3111-4bdc-91ef-09b83a4a124c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:05:22,631 [235] [DEBUG] [app] Ending request: urn:request:6b9af36d-d503-42f9-84cd-ce3420697f0a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b9af36d-d503-42f9-84cd-ce3420697f0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:05:22,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.003) gunicorn-web stdout | 2023-01-17 06:05:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:22,635 [243] [DEBUG] [app] Starting request: urn:request:62407246-a5f1-4105-afbe-9b89f71cf28d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:22,635 [239] [DEBUG] [app] Starting request: urn:request:ea477de5-6dfc-41c5-851b-60ac0a714655 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:22,635 [243] [DEBUG] [app] Ending request: urn:request:62407246-a5f1-4105-afbe-9b89f71cf28d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:62407246-a5f1-4105-afbe-9b89f71cf28d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:22,635 [239] [DEBUG] [app] Ending request: urn:request:ea477de5-6dfc-41c5-851b-60ac0a714655 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea477de5-6dfc-41c5-851b-60ac0a714655', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.000) gunicorn-web stdout | 2023-01-17 06:05:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.000) gunicorn-web stdout | 2023-01-17 06:05:22,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:22,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:22,647 [241] [DEBUG] [app] Ending request: urn:request:3446addd-4fc8-4897-9c0e-4cd093a53720 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3446addd-4fc8-4897-9c0e-4cd093a53720', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:22,647 [244] [DEBUG] [app] Ending request: urn:request:cadced52-83dc-4236-85b4-e8b02a80e597 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cadced52-83dc-4236-85b4-e8b02a80e597', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:22,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:22,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:05:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 06:05:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:39.230038+00:00 (in 16.392211 seconds) securityworker stdout | 2023-01-17 06:05:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:52 UTC)" (scheduled at 2023-01-17 06:05:22.837365+00:00) securityworker stdout | 2023-01-17 06:05:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:05:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:05:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:48.830274+00:00 (in 24.188003 seconds) exportactionlogsworker stdout | 2023-01-17 06:05:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:24 UTC)" (scheduled at 2023-01-17 06:05:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:05:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:05:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:05:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:05:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:20.242198+00:00 (in 54.907622 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:05:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:25 UTC)" (scheduled at 2023-01-17 06:05:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:05:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:05:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:25 UTC)" executed successfully builder stdout | 2023-01-17 06:05:25,444 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:25,452 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:25,452 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:25,458 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:05:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:05:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:57.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 06:05:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:57 UTC)" (scheduled at 2023-01-17 06:05:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:05:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 5, 27, 227228), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:05:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:05:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:05:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:38.735066+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 06:05:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:38 UTC)" (scheduled at 2023-01-17 06:05:28.735066+00:00) notificationworker stdout | 2023-01-17 06:05:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 28, 735945), True, datetime.datetime(2023, 1, 17, 6, 5, 28, 735945), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:05:32,221 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:05:32,445 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:05:32,682 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:05:33,552 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:05:33,561 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:05:34,796 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:05:35,468 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:35,476 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:35,476 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:35,482 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:05:35,493 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:05:35,572 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:05:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:05:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:05.848675+00:00 (in 29.999562 seconds) gcworker stdout | 2023-01-17 06:05:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:05 UTC)" (scheduled at 2023-01-17 06:05:35.848675+00:00) gcworker stdout | 2023-01-17 06:05:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725935849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:05:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:05:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:05:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:05 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:05:35,955 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:05:36,416 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:05:36,418 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:05:36,420 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:05:36,873 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:05:36,968 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:05:37,625 [239] [DEBUG] [app] Starting request: urn:request:90aa5295-91e2-46d4-ad0e-ea303d85fcec (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:37,625 [241] [DEBUG] [app] Starting request: urn:request:d9c2d0f6-426f-4600-ade1-55b840147ac9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:37,630 [232] [DEBUG] [app] Starting request: urn:request:e153a1a0-144f-4cca-9197-09db39647238 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,630 [236] [DEBUG] [app] Starting request: urn:request:5956213b-e78c-4862-a5b0-c02b1e6e5f7c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,630 [232] [DEBUG] [app] Ending request: urn:request:e153a1a0-144f-4cca-9197-09db39647238 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e153a1a0-144f-4cca-9197-09db39647238', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:05:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:05:37,631 [236] [DEBUG] [app] Ending request: urn:request:5956213b-e78c-4862-a5b0-c02b1e6e5f7c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5956213b-e78c-4862-a5b0-c02b1e6e5f7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:05:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:37,635 [244] [DEBUG] [app] Starting request: urn:request:d986cbfc-758e-4a06-8e68-ee44f6efc3ef (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:37,635 [243] [DEBUG] [app] Starting request: urn:request:b8175927-6bad-41a4-8816-befd4f21405d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:37,635 [244] [DEBUG] [app] Ending request: urn:request:d986cbfc-758e-4a06-8e68-ee44f6efc3ef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d986cbfc-758e-4a06-8e68-ee44f6efc3ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:37,636 [243] [DEBUG] [app] Ending request: urn:request:b8175927-6bad-41a4-8816-befd4f21405d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8175927-6bad-41a4-8816-befd4f21405d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:05:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:37,647 [239] [DEBUG] [app] Ending request: urn:request:90aa5295-91e2-46d4-ad0e-ea303d85fcec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90aa5295-91e2-46d4-ad0e-ea303d85fcec', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:37,647 [241] [DEBUG] [app] Ending request: urn:request:d9c2d0f6-426f-4600-ade1-55b840147ac9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9c2d0f6-426f-4600-ade1-55b840147ac9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:05:37,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:05:37,748 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:05:38,581 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:05:38,609 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:05:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:43.935629+00:00 (in 5.200167 seconds) notificationworker stdout | 2023-01-17 06:05:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:48 UTC)" (scheduled at 2023-01-17 06:05:38.735066+00:00) notificationworker stdout | 2023-01-17 06:05:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 38, 735867), True, datetime.datetime(2023, 1, 17, 6, 5, 38, 735867), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:05:38,874 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:05:38,877 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:05:38,879 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:05:38,882 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:05:38,884 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:05:38,887 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:05:38,890 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:05:38,897 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:05:38,901 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:05:38,926 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:05:38,930 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:05:38,934 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:05:38,939 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:05:38,944 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:05:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:05:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:48.738458+00:00 (in 9.707227 seconds) repositorygcworker stdout | 2023-01-17 06:05:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:39 UTC)" (scheduled at 2023-01-17 06:05:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:05:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:05:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 39, 31671), True, datetime.datetime(2023, 1, 17, 6, 5, 39, 31671), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:05:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:05:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:05:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:05:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:52.837365+00:00 (in 13.606897 seconds) securityworker stdout | 2023-01-17 06:05:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:09 UTC)" (scheduled at 2023-01-17 06:05:39.230038+00:00) securityworker stdout | 2023-01-17 06:05:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:05:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:05:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:05:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:05:39,753 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:05:40,363 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:05:41,166 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:05:41,216 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:05:41,273 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:05:41,383 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:05:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:48.735066+00:00 (in 4.798950 seconds) notificationworker stdout | 2023-01-17 06:05:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:43 UTC)" (scheduled at 2023-01-17 06:05:43.935629+00:00) notificationworker stdout | 2023-01-17 06:05:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:05:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:43 UTC)" executed successfully builder stdout | 2023-01-17 06:05:45,487 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:45,496 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:45,497 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:45,502 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:05:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:05:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:59.337413+00:00 (in 11.404522 seconds) namespacegcworker stdout | 2023-01-17 06:05:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:47 UTC)" (scheduled at 2023-01-17 06:05:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:05:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:05:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:58.735066+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 06:05:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:58 UTC)" (scheduled at 2023-01-17 06:05:48.735066+00:00) notificationworker stdout | 2023-01-17 06:05:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 48, 735911), True, datetime.datetime(2023, 1, 17, 6, 5, 48, 735911), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:05:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:05:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:39.030814+00:00 (in 50.291886 seconds) repositorygcworker stdout | 2023-01-17 06:05:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:48 UTC)" (scheduled at 2023-01-17 06:05:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:05:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:05:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:05:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:05:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:24.641816+00:00 (in 35.811062 seconds) exportactionlogsworker stdout | 2023-01-17 06:05:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:48 UTC)" (scheduled at 2023-01-17 06:05:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:05:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:05:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 48, 831126), True, datetime.datetime(2023, 1, 17, 6, 5, 48, 831126), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:05:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:05:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:05:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:05:52,625 [244] [DEBUG] [app] Starting request: urn:request:3a03b8ed-93cb-46f0-ab49-c85d513e8f2d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:52,625 [239] [DEBUG] [app] Starting request: urn:request:e7fa4c36-a18e-4b10-a70f-717e63a03710 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:05:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:52,630 [232] [DEBUG] [app] Starting request: urn:request:6ce4ffa9-1db6-419b-8947-2ffb6f98be48 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:52,630 [242] [DEBUG] [app] Starting request: urn:request:8a9975f3-b693-4d45-803a-9f7dd96a0867 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:52,630 [232] [DEBUG] [app] Ending request: urn:request:6ce4ffa9-1db6-419b-8947-2ffb6f98be48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ce4ffa9-1db6-419b-8947-2ffb6f98be48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:52,630 [242] [DEBUG] [app] Ending request: urn:request:8a9975f3-b693-4d45-803a-9f7dd96a0867 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a9975f3-b693-4d45-803a-9f7dd96a0867', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:05:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:05:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:52,635 [244] [DEBUG] [app] Starting request: urn:request:a99bdd38-b050-4b4e-864e-9bf427f05927 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:52,635 [244] [DEBUG] [app] Ending request: urn:request:a99bdd38-b050-4b4e-864e-9bf427f05927 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a99bdd38-b050-4b4e-864e-9bf427f05927', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:52,636 [241] [DEBUG] [app] Starting request: urn:request:69dc2e10-bdf0-4e80-acda-7c293007e635 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,636 [241] [DEBUG] [app] Ending request: urn:request:69dc2e10-bdf0-4e80-acda-7c293007e635 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:69dc2e10-bdf0-4e80-acda-7c293007e635', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:52,647 [244] [DEBUG] [app] Ending request: urn:request:3a03b8ed-93cb-46f0-ab49-c85d513e8f2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a03b8ed-93cb-46f0-ab49-c85d513e8f2d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:52,647 [239] [DEBUG] [app] Ending request: urn:request:e7fa4c36-a18e-4b10-a70f-717e63a03710 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7fa4c36-a18e-4b10-a70f-717e63a03710', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:52,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:05:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 06:05:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:09.230038+00:00 (in 16.392243 seconds) securityworker stdout | 2023-01-17 06:05:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:22 UTC)" (scheduled at 2023-01-17 06:05:52.837365+00:00) securityworker stdout | 2023-01-17 06:05:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:22 UTC)" executed successfully builder stdout | 2023-01-17 06:05:55,503 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:55,512 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:55,512 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:55,517 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:05:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:05:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:27.226424+00:00 (in 29.999543 seconds) buildlogsarchiver stdout | 2023-01-17 06:05:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:27 UTC)" (scheduled at 2023-01-17 06:05:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:05:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 5, 57, 227282), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:05:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:05:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:05:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:08.735066+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 06:05:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:08 UTC)" (scheduled at 2023-01-17 06:05:58.735066+00:00) notificationworker stdout | 2023-01-17 06:05:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 58, 735960), True, datetime.datetime(2023, 1, 17, 6, 5, 58, 735960), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:05:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:05:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:47.932490+00:00 (in 48.594663 seconds) namespacegcworker stdout | 2023-01-17 06:05:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:59 UTC)" (scheduled at 2023-01-17 06:05:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:05:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:05:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 59, 338250), True, datetime.datetime(2023, 1, 17, 6, 5, 59, 338250), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:05:59,347 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:05:59,347 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:05:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:06:02,237 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:06:02,455 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:06:02,707 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:06:03,558 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:06:03,580 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:06:04,812 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:06:05,517 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:06:05,527 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:05,535 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:05,535 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:05,541 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:06:05,590 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:06:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:06:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:35.848675+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 06:06:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:35 UTC)" (scheduled at 2023-01-17 06:06:05.848675+00:00) gcworker stdout | 2023-01-17 06:06:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725965849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:06:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:06:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:06:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:35 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:06:05,988 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:06:06,422 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:06:06,424 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:06:06,427 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:06:06,909 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:06:07,003 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:06:07,626 [239] [DEBUG] [app] Starting request: urn:request:5ad95d95-b98c-4c18-88be-58cdd588c7bc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:07,626 [244] [DEBUG] [app] Starting request: urn:request:5ade8683-18ae-481f-b9b8-8b354cf90ff4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:07,630 [232] [DEBUG] [app] Starting request: urn:request:13e4035a-f7e5-4d6f-889e-3c27d3b67ff2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:07,630 [242] [DEBUG] [app] Starting request: urn:request:38858f58-4949-4804-80e4-444ac33c1742 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:07,630 [232] [DEBUG] [app] Ending request: urn:request:13e4035a-f7e5-4d6f-889e-3c27d3b67ff2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13e4035a-f7e5-4d6f-889e-3c27d3b67ff2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:06:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:06:07,631 [242] [DEBUG] [app] Ending request: urn:request:38858f58-4949-4804-80e4-444ac33c1742 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:38858f58-4949-4804-80e4-444ac33c1742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:06:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:06:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:07,635 [243] [DEBUG] [app] Starting request: urn:request:815c08cf-4597-4604-b7fb-fcd817583432 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:07,635 [241] [DEBUG] [app] Starting request: urn:request:564ba06f-6c81-4e1b-80e8-ca340f24c4c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:07,635 [243] [DEBUG] [app] Ending request: urn:request:815c08cf-4597-4604-b7fb-fcd817583432 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:815c08cf-4597-4604-b7fb-fcd817583432', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:07,636 [241] [DEBUG] [app] Ending request: urn:request:564ba06f-6c81-4e1b-80e8-ca340f24c4c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:564ba06f-6c81-4e1b-80e8-ca340f24c4c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:07,646 [239] [DEBUG] [app] Ending request: urn:request:5ad95d95-b98c-4c18-88be-58cdd588c7bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ad95d95-b98c-4c18-88be-58cdd588c7bc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:07,646 [244] [DEBUG] [app] Ending request: urn:request:5ade8683-18ae-481f-b9b8-8b354cf90ff4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ade8683-18ae-481f-b9b8-8b354cf90ff4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:07,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:07,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:06:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.021 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.021 47 0.021) chunkcleanupworker stdout | 2023-01-17 06:06:07,784 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:06:08,590 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:06:08,614 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:06:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:18.735066+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 06:06:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:18 UTC)" (scheduled at 2023-01-17 06:06:08.735066+00:00) notificationworker stdout | 2023-01-17 06:06:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 8, 735909), True, datetime.datetime(2023, 1, 17, 6, 6, 8, 735909), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:06:08,884 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:06:08,886 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:06:08,889 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:06:08,892 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:06:08,894 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:06:08,897 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:06:08,899 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:06:08,906 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:06:08,908 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:06:08,931 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:06:08,936 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:06:08,941 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:06:08,945 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:06:08,949 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:06:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:22.837365+00:00 (in 13.606901 seconds) securityworker stdout | 2023-01-17 06:06:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:39 UTC)" (scheduled at 2023-01-17 06:06:09.230038+00:00) securityworker stdout | 2023-01-17 06:06:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:06:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:06:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:06:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:06:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:06:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:06:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:06:09,765 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:06:10,382 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:06:11,202 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:06:11,251 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:06:11,308 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:06:11,419 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:06:15,551 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:15,560 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:15,560 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:15,566 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:28.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 06:06:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:28 UTC)" (scheduled at 2023-01-17 06:06:18.735066+00:00) notificationworker stdout | 2023-01-17 06:06:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 18, 735921), True, datetime.datetime(2023, 1, 17, 6, 6, 18, 735921), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:06:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:06:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:25.334138+00:00 (in 5.091472 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:06:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:20 UTC)" (scheduled at 2023-01-17 06:06:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:06:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:06:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 20, 243108), True, datetime.datetime(2023, 1, 17, 6, 6, 20, 243108), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:06:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:06:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:06:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:06:22,625 [244] [DEBUG] [app] Starting request: urn:request:f435d11a-cbad-4a5d-a50c-8aee0777abdc (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:22,626 [243] [DEBUG] [app] Starting request: urn:request:3845298f-fee1-43f4-890c-0f1ee9c9816a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:22,630 [242] [DEBUG] [app] Starting request: urn:request:59ba1614-1fc9-4b06-9c9b-8f22c1cbd75f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,631 [237] [DEBUG] [app] Starting request: urn:request:5971ce40-7331-4d8d-962a-a00715e3de7c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,631 [242] [DEBUG] [app] Ending request: urn:request:59ba1614-1fc9-4b06-9c9b-8f22c1cbd75f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59ba1614-1fc9-4b06-9c9b-8f22c1cbd75f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,631 [237] [DEBUG] [app] Ending request: urn:request:5971ce40-7331-4d8d-962a-a00715e3de7c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5971ce40-7331-4d8d-962a-a00715e3de7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:06:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:06:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:22,635 [239] [DEBUG] [app] Starting request: urn:request:3ab6e7cd-5c69-4f26-aa7e-f1402bd1cf5f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:22,636 [244] [DEBUG] [app] Starting request: urn:request:0a7222f2-fb37-4c2a-8f10-430b1316c35b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:22,636 [239] [DEBUG] [app] Ending request: urn:request:3ab6e7cd-5c69-4f26-aa7e-f1402bd1cf5f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ab6e7cd-5c69-4f26-aa7e-f1402bd1cf5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:22,636 [244] [DEBUG] [app] Ending request: urn:request:0a7222f2-fb37-4c2a-8f10-430b1316c35b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a7222f2-fb37-4c2a-8f10-430b1316c35b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:22,648 [243] [DEBUG] [app] Ending request: urn:request:3845298f-fee1-43f4-890c-0f1ee9c9816a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3845298f-fee1-43f4-890c-0f1ee9c9816a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:22,648 [244] [DEBUG] [app] Ending request: urn:request:f435d11a-cbad-4a5d-a50c-8aee0777abdc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f435d11a-cbad-4a5d-a50c-8aee0777abdc', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 06:06:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:06:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:39.230038+00:00 (in 16.392220 seconds) securityworker stdout | 2023-01-17 06:06:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:52 UTC)" (scheduled at 2023-01-17 06:06:22.837365+00:00) securityworker stdout | 2023-01-17 06:06:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:06:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:06:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:44.641497+00:00 (in 19.999192 seconds) exportactionlogsworker stdout | 2023-01-17 06:06:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:24 UTC)" (scheduled at 2023-01-17 06:06:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:06:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:06:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:06:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:06:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:39.333860+00:00 (in 13.999235 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:06:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:25 UTC)" (scheduled at 2023-01-17 06:06:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:06:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:06:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:25 UTC)" executed successfully builder stdout | 2023-01-17 06:06:25,576 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:25,585 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:25,585 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:25,591 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:06:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:06:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:57.226424+00:00 (in 29.999508 seconds) buildlogsarchiver stdout | 2023-01-17 06:06:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:57 UTC)" (scheduled at 2023-01-17 06:06:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:06:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 6, 27, 227333), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:06:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:06:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:06:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:38.735066+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 06:06:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:38 UTC)" (scheduled at 2023-01-17 06:06:28.735066+00:00) notificationworker stdout | 2023-01-17 06:06:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 28, 735951), True, datetime.datetime(2023, 1, 17, 6, 6, 28, 735951), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:06:32,243 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:06:32,490 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:06:32,742 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:06:33,564 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:06:33,614 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:06:34,848 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:06:35,541 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:06:35,597 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 06:06:35,601 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:35,610 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:35,610 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:35,615 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:06:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:06:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:05.848675+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 06:06:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:05 UTC)" (scheduled at 2023-01-17 06:06:35.848675+00:00) gcworker stdout | 2023-01-17 06:06:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725995849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:06:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:06:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:06:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:05 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:06:36,023 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:06:36,430 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:06:36,432 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:06:36,435 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:06:36,926 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:06:37,038 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:06:37,625 [239] [DEBUG] [app] Starting request: urn:request:59c30f35-f5fd-475b-a14a-fe3b6519f1e8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:37,625 [244] [DEBUG] [app] Starting request: urn:request:24c93439-1507-4e9d-b336-2278e53d5397 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:37,630 [232] [DEBUG] [app] Starting request: urn:request:57d652f5-a029-49a9-ba08-153dd56d7422 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:37,630 [242] [DEBUG] [app] Starting request: urn:request:40cf2b7e-083e-46d4-999b-2c79c1d275e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:37,631 [232] [DEBUG] [app] Ending request: urn:request:57d652f5-a029-49a9-ba08-153dd56d7422 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57d652f5-a029-49a9-ba08-153dd56d7422', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:06:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:06:37,631 [242] [DEBUG] [app] Ending request: urn:request:40cf2b7e-083e-46d4-999b-2c79c1d275e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40cf2b7e-083e-46d4-999b-2c79c1d275e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:06:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:06:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:37,635 [241] [DEBUG] [app] Starting request: urn:request:edc722a6-ce92-469f-9831-52a57afd5c74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:37,635 [243] [DEBUG] [app] Starting request: urn:request:8e207e8c-7adf-4f5e-816f-075f6ca23cfd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:37,636 [241] [DEBUG] [app] Ending request: urn:request:edc722a6-ce92-469f-9831-52a57afd5c74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:edc722a6-ce92-469f-9831-52a57afd5c74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:37,636 [243] [DEBUG] [app] Ending request: urn:request:8e207e8c-7adf-4f5e-816f-075f6ca23cfd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e207e8c-7adf-4f5e-816f-075f6ca23cfd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:37,647 [239] [DEBUG] [app] Ending request: urn:request:59c30f35-f5fd-475b-a14a-fe3b6519f1e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:59c30f35-f5fd-475b-a14a-fe3b6519f1e8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:37,647 [244] [DEBUG] [app] Ending request: urn:request:24c93439-1507-4e9d-b336-2278e53d5397 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24c93439-1507-4e9d-b336-2278e53d5397', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:06:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:06:37,798 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:06:38,625 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:06:38,628 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:06:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:43.935629+00:00 (in 5.200112 seconds) notificationworker stdout | 2023-01-17 06:06:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:48 UTC)" (scheduled at 2023-01-17 06:06:38.735066+00:00) notificationworker stdout | 2023-01-17 06:06:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 38, 735920), True, datetime.datetime(2023, 1, 17, 6, 6, 38, 735920), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:06:38,894 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:06:38,896 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:06:38,899 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:06:38,903 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:06:38,905 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:06:38,908 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:06:38,910 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:06:38,914 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:06:38,917 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:06:38,937 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:06:38,941 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:06:38,946 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:06:38,950 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:06:38,954 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:06:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:06:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:48.738458+00:00 (in 9.707174 seconds) repositorygcworker stdout | 2023-01-17 06:06:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:39 UTC)" (scheduled at 2023-01-17 06:06:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:06:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:06:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 39, 31720), True, datetime.datetime(2023, 1, 17, 6, 6, 39, 31720), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:06:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:06:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:06:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:06:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:52.837365+00:00 (in 13.606878 seconds) securityworker stdout | 2023-01-17 06:06:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:09 UTC)" (scheduled at 2023-01-17 06:06:39.230038+00:00) securityworker stdout | 2023-01-17 06:06:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:06:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:06:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:06:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:06:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:06:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:20.242198+00:00 (in 40.907892 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:06:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:39 UTC)" (scheduled at 2023-01-17 06:06:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:06:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 6, 39, 334685), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:06:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 39, 334685), True, datetime.datetime(2023, 1, 17, 6, 6, 39, 334685), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:06:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 39, 334685), True, datetime.datetime(2023, 1, 17, 6, 6, 39, 334685), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 6, 39, 334685), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:06:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:06:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:06:39,788 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:06:40,405 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:06:41,208 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:06:41,281 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:06:41,327 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:06:41,429 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:06:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:48.735066+00:00 (in 4.798976 seconds) notificationworker stdout | 2023-01-17 06:06:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:43 UTC)" (scheduled at 2023-01-17 06:06:43.935629+00:00) notificationworker stdout | 2023-01-17 06:06:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:06:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:06:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:06:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:48.830274+00:00 (in 4.188307 seconds) exportactionlogsworker stdout | 2023-01-17 06:06:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:44 UTC)" (scheduled at 2023-01-17 06:06:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 06:06:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 6, 44, 642272), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:06:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 44, 642272), True, datetime.datetime(2023, 1, 17, 6, 6, 44, 642272), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:06:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 44, 642272), True, datetime.datetime(2023, 1, 17, 6, 6, 44, 642272), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 6, 44, 642272), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:06:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:06:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:44 UTC)" executed successfully builder stdout | 2023-01-17 06:06:45,625 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:45,641 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:45,641 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:45,647 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:06:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:06:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:59.337413+00:00 (in 11.404446 seconds) namespacegcworker stdout | 2023-01-17 06:06:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:47 UTC)" (scheduled at 2023-01-17 06:06:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:06:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:06:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:58.735066+00:00 (in 9.999503 seconds) notificationworker stdout | 2023-01-17 06:06:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:58 UTC)" (scheduled at 2023-01-17 06:06:48.735066+00:00) notificationworker stdout | 2023-01-17 06:06:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 48, 735931), True, datetime.datetime(2023, 1, 17, 6, 6, 48, 735931), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:06:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:06:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:04.738062+00:00 (in 15.999133 seconds) repositorygcworker stdout | 2023-01-17 06:06:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:48 UTC)" (scheduled at 2023-01-17 06:06:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:06:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:06:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:06:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:06:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:24.641816+00:00 (in 35.811119 seconds) exportactionlogsworker stdout | 2023-01-17 06:06:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:48 UTC)" (scheduled at 2023-01-17 06:06:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:06:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:06:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 48, 831069), True, datetime.datetime(2023, 1, 17, 6, 6, 48, 831069), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:06:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:06:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:06:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:06:52,625 [243] [DEBUG] [app] Starting request: urn:request:afc203c6-2647-4bcc-a4d5-6a797f87f6e2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:52,625 [244] [DEBUG] [app] Starting request: urn:request:ddb85d39-2925-47d9-abd1-8ddf2de22577 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:06:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:52,631 [242] [DEBUG] [app] Starting request: urn:request:3f274c76-22fc-4c28-9636-8ed08ac040a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:52,631 [232] [DEBUG] [app] Starting request: urn:request:974d320d-547a-491d-af34-fca83237361e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:52,631 [242] [DEBUG] [app] Ending request: urn:request:3f274c76-22fc-4c28-9636-8ed08ac040a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f274c76-22fc-4c28-9636-8ed08ac040a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:06:52,632 [232] [DEBUG] [app] Ending request: urn:request:974d320d-547a-491d-af34-fca83237361e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:974d320d-547a-491d-af34-fca83237361e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:06:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:06:52,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:52,636 [239] [DEBUG] [app] Starting request: urn:request:0360e541-c967-4bcf-a1a2-1b6f3c150665 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:52,636 [239] [DEBUG] [app] Ending request: urn:request:0360e541-c967-4bcf-a1a2-1b6f3c150665 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0360e541-c967-4bcf-a1a2-1b6f3c150665', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:52,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:52,637 [241] [DEBUG] [app] Starting request: urn:request:0f03def6-4291-403b-977d-be856d78129f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:52,638 [241] [DEBUG] [app] Ending request: urn:request:0f03def6-4291-403b-977d-be856d78129f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0f03def6-4291-403b-977d-be856d78129f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:52,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:52,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:52,639 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:52,648 [244] [DEBUG] [app] Ending request: urn:request:ddb85d39-2925-47d9-abd1-8ddf2de22577 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddb85d39-2925-47d9-abd1-8ddf2de22577', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:06:52,649 [243] [DEBUG] [app] Ending request: urn:request:afc203c6-2647-4bcc-a4d5-6a797f87f6e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:afc203c6-2647-4bcc-a4d5-6a797f87f6e2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:52,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) securityworker stdout | 2023-01-17 06:06:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:09.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 06:06:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:22 UTC)" (scheduled at 2023-01-17 06:06:52.837365+00:00) securityworker stdout | 2023-01-17 06:06:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:52,850 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:52,852 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:52,852 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:22 UTC)" executed successfully builder stdout | 2023-01-17 06:06:55,657 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:55,666 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:55,666 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:55,672 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:06:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:06:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:27.226424+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 06:06:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:27 UTC)" (scheduled at 2023-01-17 06:06:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:06:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 6, 57, 227529), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:06:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:06:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:06:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:08.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 06:06:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:08 UTC)" (scheduled at 2023-01-17 06:06:58.735066+00:00) notificationworker stdout | 2023-01-17 06:06:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 58, 735871), True, datetime.datetime(2023, 1, 17, 6, 6, 58, 735871), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:06:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:06:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:47.932490+00:00 (in 48.594528 seconds) namespacegcworker stdout | 2023-01-17 06:06:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:59 UTC)" (scheduled at 2023-01-17 06:06:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:06:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:06:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 59, 338481), True, datetime.datetime(2023, 1, 17, 6, 6, 59, 338481), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:06:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:06:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:06:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:07:02,264 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:07:02,525 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:07:02,752 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:07:03,600 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:07:03,630 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:07:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:07:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:39.030814+00:00 (in 34.292293 seconds) repositorygcworker stdout | 2023-01-17 06:07:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:12:04 UTC)" (scheduled at 2023-01-17 06:07:04.738062+00:00) repositorygcworker stdout | 2023-01-17 06:07:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 7, 4, 738910), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:07:04,747 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 7, 4, 738910), True, datetime.datetime(2023, 1, 17, 6, 7, 4, 738910), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:07:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 7, 4, 738910), True, datetime.datetime(2023, 1, 17, 6, 7, 4, 738910), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 7, 4, 738910), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:07:04,751 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:07:04,751 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:12:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:07:04,854 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:07:05,557 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:07:05,632 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 06:07:05,682 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:05,690 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:05,690 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:05,696 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:07:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:07:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:35.848675+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 06:07:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:35 UTC)" (scheduled at 2023-01-17 06:07:05.848675+00:00) gcworker stdout | 2023-01-17 06:07:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726025849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:07:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:07:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:07:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:35 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:07:06,058 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:07:06,437 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:07:06,440 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:07:06,443 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:07:06,954 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:07:07,054 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:07:07,625 [239] [DEBUG] [app] Starting request: urn:request:eb231df5-3e83-4862-9d73-f47767e6520b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:07,625 [244] [DEBUG] [app] Starting request: urn:request:7fa5ae4a-bc80-4041-8063-56ea8945b71a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:07,630 [242] [DEBUG] [app] Starting request: urn:request:b67a0a3e-eb0e-4850-ae7f-ba0d5dabe988 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:07,630 [232] [DEBUG] [app] Starting request: urn:request:3e75e702-49ba-4261-9ba7-67485d9c34a9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:07,630 [242] [DEBUG] [app] Ending request: urn:request:b67a0a3e-eb0e-4850-ae7f-ba0d5dabe988 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b67a0a3e-eb0e-4850-ae7f-ba0d5dabe988', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:07,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:07:07,630 [232] [DEBUG] [app] Ending request: urn:request:3e75e702-49ba-4261-9ba7-67485d9c34a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e75e702-49ba-4261-9ba7-67485d9c34a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:07:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:07:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:07,635 [243] [DEBUG] [app] Starting request: urn:request:98ad9dad-863e-4ccc-9d15-898b46553720 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:07,635 [239] [DEBUG] [app] Starting request: urn:request:07b8c935-68dc-4586-9e3f-f80653eb17ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:07,635 [243] [DEBUG] [app] Ending request: urn:request:98ad9dad-863e-4ccc-9d15-898b46553720 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98ad9dad-863e-4ccc-9d15-898b46553720', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:07,635 [239] [DEBUG] [app] Ending request: urn:request:07b8c935-68dc-4586-9e3f-f80653eb17ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:07b8c935-68dc-4586-9e3f-f80653eb17ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:07:07,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:07,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:07,635 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:07,640 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:07,640 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:07,646 [244] [DEBUG] [app] Ending request: urn:request:7fa5ae4a-bc80-4041-8063-56ea8945b71a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fa5ae4a-bc80-4041-8063-56ea8945b71a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:07,646 [239] [DEBUG] [app] Ending request: urn:request:eb231df5-3e83-4862-9d73-f47767e6520b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb231df5-3e83-4862-9d73-f47767e6520b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:07,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:07,646 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 06:07:07,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:07,646 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:07:07,821 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:07:08,660 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:07:08,662 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:07:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:18.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 06:07:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:18 UTC)" (scheduled at 2023-01-17 06:07:08.735066+00:00) notificationworker stdout | 2023-01-17 06:07:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 8, 735878), True, datetime.datetime(2023, 1, 17, 6, 7, 8, 735878), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:07:08,902 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:07:08,904 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:07:08,909 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:07:08,911 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:07:08,914 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:07:08,916 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:07:08,919 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:07:08,922 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:07:08,925 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:07:08,943 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:07:08,946 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:07:08,952 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:07:08,956 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:07:08,960 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:07:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:22.837365+00:00 (in 13.606769 seconds) securityworker stdout | 2023-01-17 06:07:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:39 UTC)" (scheduled at 2023-01-17 06:07:09.230038+00:00) securityworker stdout | 2023-01-17 06:07:09,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:07:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:07:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:07:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:07:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:07:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:07:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:07:09,795 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:07:10,423 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:07:11,244 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:07:11,291 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:07:11,362 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:07:11,445 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:07:15,706 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:15,714 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:15,714 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:15,720 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:28.735066+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 06:07:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:28 UTC)" (scheduled at 2023-01-17 06:07:18.735066+00:00) notificationworker stdout | 2023-01-17 06:07:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 18, 735911), True, datetime.datetime(2023, 1, 17, 6, 7, 18, 735911), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:07:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:07:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:25.334138+00:00 (in 5.091380 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:07:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:20 UTC)" (scheduled at 2023-01-17 06:07:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:07:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:07:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 20, 243191), True, datetime.datetime(2023, 1, 17, 6, 7, 20, 243191), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:07:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:07:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:07:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:07:22,625 [241] [DEBUG] [app] Starting request: urn:request:507552bc-8799-4400-820a-189cb316a0cb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,625 [239] [DEBUG] [app] Starting request: urn:request:71c3e56c-bb10-41a5-9c65-643282daa685 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:22,630 [242] [DEBUG] [app] Starting request: urn:request:0dae8493-00d0-4346-b08d-e6b08d67f377 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:22,630 [236] [DEBUG] [app] Starting request: urn:request:ab476e95-d235-4fe9-9c34-6018e0027161 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:22,630 [242] [DEBUG] [app] Ending request: urn:request:0dae8493-00d0-4346-b08d-e6b08d67f377 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0dae8493-00d0-4346-b08d-e6b08d67f377', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:07:22,631 [236] [DEBUG] [app] Ending request: urn:request:ab476e95-d235-4fe9-9c34-6018e0027161 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab476e95-d235-4fe9-9c34-6018e0027161', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:07:22,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:07:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:22,635 [244] [DEBUG] [app] Starting request: urn:request:13811018-862c-4005-abba-5b9efed011db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,635 [243] [DEBUG] [app] Starting request: urn:request:fa50d74c-fb96-46ed-9860-83c62c8e49b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,635 [244] [DEBUG] [app] Ending request: urn:request:13811018-862c-4005-abba-5b9efed011db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:13811018-862c-4005-abba-5b9efed011db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:22,635 [243] [DEBUG] [app] Ending request: urn:request:fa50d74c-fb96-46ed-9860-83c62c8e49b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa50d74c-fb96-46ed-9860-83c62c8e49b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:22,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:22,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:22,646 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:22,653 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:22,655 [241] [DEBUG] [app] Ending request: urn:request:507552bc-8799-4400-820a-189cb316a0cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:507552bc-8799-4400-820a-189cb316a0cb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:22,655 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.032) gunicorn-web stdout | 2023-01-17 06:07:22,655 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:22,658 [239] [DEBUG] [app] Ending request: urn:request:71c3e56c-bb10-41a5-9c65-643282daa685 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71c3e56c-bb10-41a5-9c65-643282daa685', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:22,658 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.034) gunicorn-web stdout | 2023-01-17 06:07:22,659 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:07:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:39.230038+00:00 (in 16.392189 seconds) securityworker stdout | 2023-01-17 06:07:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:52 UTC)" (scheduled at 2023-01-17 06:07:22.837365+00:00) securityworker stdout | 2023-01-17 06:07:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:07:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:07:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:48.830274+00:00 (in 24.188059 seconds) exportactionlogsworker stdout | 2023-01-17 06:07:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:24 UTC)" (scheduled at 2023-01-17 06:07:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:07:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:07:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:07:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:07:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:20.242198+00:00 (in 54.907600 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:07:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:25 UTC)" (scheduled at 2023-01-17 06:07:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:07:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:07:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:25 UTC)" executed successfully builder stdout | 2023-01-17 06:07:25,726 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:25,734 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:25,734 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:25,740 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:07:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:07:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:57.226424+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2023-01-17 06:07:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:57 UTC)" (scheduled at 2023-01-17 06:07:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:07:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 7, 27, 227202), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:07:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:07:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:07:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:38.735066+00:00 (in 9.999582 seconds) notificationworker stdout | 2023-01-17 06:07:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:38 UTC)" (scheduled at 2023-01-17 06:07:28.735066+00:00) notificationworker stdout | 2023-01-17 06:07:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 28, 735918), True, datetime.datetime(2023, 1, 17, 6, 7, 28, 735918), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:07:32,299 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:07:32,560 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:07:32,786 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:07:33,633 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:07:33,665 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:07:34,866 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:07:35,580 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:07:35,645 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 06:07:35,750 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:35,759 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:35,759 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:35,765 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:07:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:07:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:05.848675+00:00 (in 29.999570 seconds) gcworker stdout | 2023-01-17 06:07:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:05 UTC)" (scheduled at 2023-01-17 06:07:35.848675+00:00) gcworker stdout | 2023-01-17 06:07:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726055849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:07:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:07:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:07:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:05 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:07:36,095 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:07:36,445 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:07:36,447 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:07:36,450 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:07:36,968 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:07:37,090 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:07:37,626 [243] [DEBUG] [app] Starting request: urn:request:a6fd111e-2659-4da9-ac7f-be37361bd768 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:37,626 [239] [DEBUG] [app] Starting request: urn:request:caefac21-5962-4fdd-941e-9489e32196ae (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:37,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,628 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:37,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:37,632 [242] [DEBUG] [app] Starting request: urn:request:654a4f34-f828-408a-a1f3-66e491db8978 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,632 [232] [DEBUG] [app] Starting request: urn:request:31f8b8a5-0b44-4385-ae58-7946a72d8e0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,632 [242] [DEBUG] [app] Ending request: urn:request:654a4f34-f828-408a-a1f3-66e491db8978 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:654a4f34-f828-408a-a1f3-66e491db8978', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:07:37,633 [232] [DEBUG] [app] Ending request: urn:request:31f8b8a5-0b44-4385-ae58-7946a72d8e0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31f8b8a5-0b44-4385-ae58-7946a72d8e0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:37,633 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:07:37,633 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:07:37,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:37,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,635 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,637 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:37,637 [239] [DEBUG] [app] Starting request: urn:request:ac771c10-2437-4db4-b696-4786afa7edeb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:37,638 [239] [DEBUG] [app] Ending request: urn:request:ac771c10-2437-4db4-b696-4786afa7edeb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac771c10-2437-4db4-b696-4786afa7edeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:37,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:37,639 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:37,639 [244] [DEBUG] [app] Starting request: urn:request:f79094a2-7888-4213-979a-d99136528114 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:37,640 [244] [DEBUG] [app] Ending request: urn:request:f79094a2-7888-4213-979a-d99136528114 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f79094a2-7888-4213-979a-d99136528114', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:37,640 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:07:37,640 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:37,641 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:37,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:37,645 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:37,649 [243] [DEBUG] [app] Ending request: urn:request:a6fd111e-2659-4da9-ac7f-be37361bd768 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6fd111e-2659-4da9-ac7f-be37361bd768', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:37,649 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:07:37,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:37,650 [239] [DEBUG] [app] Ending request: urn:request:caefac21-5962-4fdd-941e-9489e32196ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:caefac21-5962-4fdd-941e-9489e32196ae', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:37,650 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:37,651 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.025) chunkcleanupworker stdout | 2023-01-17 06:07:37,836 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:07:38,697 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:07:38,699 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:07:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:43.935629+00:00 (in 5.200163 seconds) notificationworker stdout | 2023-01-17 06:07:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:48 UTC)" (scheduled at 2023-01-17 06:07:38.735066+00:00) notificationworker stdout | 2023-01-17 06:07:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 38, 735818), True, datetime.datetime(2023, 1, 17, 6, 7, 38, 735818), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:07:38,913 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:07:38,915 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:07:38,918 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:07:38,920 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:07:38,923 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:07:38,926 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:07:38,929 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:07:38,931 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:07:38,934 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:07:38,950 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:07:38,953 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:07:38,957 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:07:38,961 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:07:38,967 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:07:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:07:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:48.738458+00:00 (in 9.707206 seconds) repositorygcworker stdout | 2023-01-17 06:07:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:39 UTC)" (scheduled at 2023-01-17 06:07:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:07:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:07:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 39, 31641), True, datetime.datetime(2023, 1, 17, 6, 7, 39, 31641), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:07:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:07:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:07:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:07:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:52.837365+00:00 (in 13.606916 seconds) securityworker stdout | 2023-01-17 06:07:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:09 UTC)" (scheduled at 2023-01-17 06:07:39.230038+00:00) securityworker stdout | 2023-01-17 06:07:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:07:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:07:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:07:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:07:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:07:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:07:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:07:39,831 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:07:40,458 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:07:41,278 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:07:41,317 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:07:41,377 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:07:41,480 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:07:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:48.735066+00:00 (in 4.798987 seconds) notificationworker stdout | 2023-01-17 06:07:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:43 UTC)" (scheduled at 2023-01-17 06:07:43.935629+00:00) notificationworker stdout | 2023-01-17 06:07:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:07:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:43 UTC)" executed successfully builder stdout | 2023-01-17 06:07:45,775 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:45,783 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:45,783 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:45,789 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:07:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:07:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:59.337413+00:00 (in 11.404369 seconds) namespacegcworker stdout | 2023-01-17 06:07:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:47 UTC)" (scheduled at 2023-01-17 06:07:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:07:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:07:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:58.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 06:07:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:58 UTC)" (scheduled at 2023-01-17 06:07:48.735066+00:00) notificationworker stdout | 2023-01-17 06:07:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 48, 735882), True, datetime.datetime(2023, 1, 17, 6, 7, 48, 735882), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:07:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:07:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:39.030814+00:00 (in 50.291914 seconds) repositorygcworker stdout | 2023-01-17 06:07:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:48 UTC)" (scheduled at 2023-01-17 06:07:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:07:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:07:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:07:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:07:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:24.641816+00:00 (in 35.811123 seconds) exportactionlogsworker stdout | 2023-01-17 06:07:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:48 UTC)" (scheduled at 2023-01-17 06:07:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:07:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:07:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 48, 831054), True, datetime.datetime(2023, 1, 17, 6, 7, 48, 831054), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:07:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:07:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:07:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:07:52,625 [239] [DEBUG] [app] Starting request: urn:request:9def8622-4063-4fe1-94e2-61c3283b22e8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:52,626 [244] [DEBUG] [app] Starting request: urn:request:1641c930-4b89-4d80-864e-850072de5fc6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:07:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:52,630 [242] [DEBUG] [app] Starting request: urn:request:ff1f9eb6-2639-4494-8989-029504357b57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:52,631 [232] [DEBUG] [app] Starting request: urn:request:34988dc0-8c7c-4e95-ad5c-052dd06db3dc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:52,631 [242] [DEBUG] [app] Ending request: urn:request:ff1f9eb6-2639-4494-8989-029504357b57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff1f9eb6-2639-4494-8989-029504357b57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:07:52,631 [232] [DEBUG] [app] Ending request: urn:request:34988dc0-8c7c-4e95-ad5c-052dd06db3dc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34988dc0-8c7c-4e95-ad5c-052dd06db3dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:07:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:07:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:52,635 [243] [DEBUG] [app] Starting request: urn:request:5a83abbc-8bbb-48a5-8423-712ba24d912c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:52,636 [241] [DEBUG] [app] Starting request: urn:request:7aeb1b3f-b92c-4fd7-8025-fa78eb1cbe49 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:52,636 [243] [DEBUG] [app] Ending request: urn:request:5a83abbc-8bbb-48a5-8423-712ba24d912c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a83abbc-8bbb-48a5-8423-712ba24d912c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.003) gunicorn-web stdout | 2023-01-17 06:07:52,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:52,636 [241] [DEBUG] [app] Ending request: urn:request:7aeb1b3f-b92c-4fd7-8025-fa78eb1cbe49 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7aeb1b3f-b92c-4fd7-8025-fa78eb1cbe49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:52,647 [239] [DEBUG] [app] Ending request: urn:request:9def8622-4063-4fe1-94e2-61c3283b22e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9def8622-4063-4fe1-94e2-61c3283b22e8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:52,647 [244] [DEBUG] [app] Ending request: urn:request:1641c930-4b89-4d80-864e-850072de5fc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1641c930-4b89-4d80-864e-850072de5fc6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:52,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:07:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:52,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) securityworker stdout | 2023-01-17 06:07:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:09.230038+00:00 (in 16.392218 seconds) securityworker stdout | 2023-01-17 06:07:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:22 UTC)" (scheduled at 2023-01-17 06:07:52.837365+00:00) securityworker stdout | 2023-01-17 06:07:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:22 UTC)" executed successfully builder stdout | 2023-01-17 06:07:55,799 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:55,807 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:55,807 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:55,813 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:07:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:07:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:27.226424+00:00 (in 29.999561 seconds) buildlogsarchiver stdout | 2023-01-17 06:07:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:27 UTC)" (scheduled at 2023-01-17 06:07:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:07:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 7, 57, 227193), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:07:57,243 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:07:57,243 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:07:57,243 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:08.735066+00:00 (in 9.999562 seconds) notificationworker stdout | 2023-01-17 06:07:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:08 UTC)" (scheduled at 2023-01-17 06:07:58.735066+00:00) notificationworker stdout | 2023-01-17 06:07:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 58, 735886), True, datetime.datetime(2023, 1, 17, 6, 7, 58, 735886), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:07:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:07:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:47.932490+00:00 (in 48.594573 seconds) namespacegcworker stdout | 2023-01-17 06:07:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:59 UTC)" (scheduled at 2023-01-17 06:07:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:07:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:07:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 59, 338384), True, datetime.datetime(2023, 1, 17, 6, 7, 59, 338384), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:07:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:07:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:07:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:08:02,305 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:08:02,595 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:08:02,791 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:08:03,664 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:08:03,697 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:08:04,886 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:08:05,617 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:08:05,653 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 06:08:05,823 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:05,832 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:05,832 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:05,838 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:08:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:08:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:35.848675+00:00 (in 29.999578 seconds) gcworker stdout | 2023-01-17 06:08:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:35 UTC)" (scheduled at 2023-01-17 06:08:05.848675+00:00) gcworker stdout | 2023-01-17 06:08:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726085849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:08:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:08:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:08:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:35 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:08:06,131 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:08:06,452 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:08:06,454 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:08:06,456 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:08:06,983 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:08:07,126 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:08:07,625 [241] [DEBUG] [app] Starting request: urn:request:0b823e44-0fb0-4bd1-9441-1ba58803fd3c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:07,625 [243] [DEBUG] [app] Starting request: urn:request:44d11639-5716-4591-bdd3-5fc78b3a9898 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:07,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:07,630 [232] [DEBUG] [app] Starting request: urn:request:53391a6d-76ae-4378-8f67-e68a0580adab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:07,630 [242] [DEBUG] [app] Starting request: urn:request:5669eb34-bd5a-4449-ba01-0d83ef09edc8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:07,631 [232] [DEBUG] [app] Ending request: urn:request:53391a6d-76ae-4378-8f67-e68a0580adab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:53391a6d-76ae-4378-8f67-e68a0580adab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:08:07,631 [242] [DEBUG] [app] Ending request: urn:request:5669eb34-bd5a-4449-ba01-0d83ef09edc8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5669eb34-bd5a-4449-ba01-0d83ef09edc8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:08:07,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:07,635 [244] [DEBUG] [app] Starting request: urn:request:00a796a5-3d4b-412e-9ec7-b31b7f23faf3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:07,635 [243] [DEBUG] [app] Starting request: urn:request:3f452a22-d514-46c6-a830-4f1589fc1e60 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:07,636 [243] [DEBUG] [app] Ending request: urn:request:3f452a22-d514-46c6-a830-4f1589fc1e60 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f452a22-d514-46c6-a830-4f1589fc1e60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:07,636 [244] [DEBUG] [app] Ending request: urn:request:00a796a5-3d4b-412e-9ec7-b31b7f23faf3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00a796a5-3d4b-412e-9ec7-b31b7f23faf3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:08:07,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:07,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:07,647 [243] [DEBUG] [app] Ending request: urn:request:44d11639-5716-4591-bdd3-5fc78b3a9898 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44d11639-5716-4591-bdd3-5fc78b3a9898', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:07,647 [241] [DEBUG] [app] Ending request: urn:request:0b823e44-0fb0-4bd1-9441-1ba58803fd3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b823e44-0fb0-4bd1-9441-1ba58803fd3c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:08:07,647 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:08:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:08:07,845 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} logrotateworker stdout | 2023-01-17 06:08:08,732 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:08:08,734 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:08:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:18.735066+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 06:08:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:18 UTC)" (scheduled at 2023-01-17 06:08:08.735066+00:00) notificationworker stdout | 2023-01-17 06:08:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 8, 735852), True, datetime.datetime(2023, 1, 17, 6, 8, 8, 735852), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:08:08,924 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:08:08,926 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:08:08,929 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:08:08,932 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:08:08,934 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:08:08,937 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:08:08,939 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:08:08,942 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:08:08,944 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:08:08,956 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:08:08,960 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:08:08,963 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:08:08,967 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:08:08,972 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:08:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:22.837365+00:00 (in 13.606869 seconds) securityworker stdout | 2023-01-17 06:08:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:39 UTC)" (scheduled at 2023-01-17 06:08:09.230038+00:00) securityworker stdout | 2023-01-17 06:08:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:08:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:08:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:08:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:08:09,861 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:08:10,475 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:08:11,305 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:08:11,344 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:08:11,400 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:08:11,513 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:08:15,848 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:15,856 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:15,856 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:15,862 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:28.735066+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 06:08:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:28 UTC)" (scheduled at 2023-01-17 06:08:18.735066+00:00) notificationworker stdout | 2023-01-17 06:08:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 18, 735923), True, datetime.datetime(2023, 1, 17, 6, 8, 18, 735923), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:08:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:08:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:25.334138+00:00 (in 5.091507 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:08:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:20 UTC)" (scheduled at 2023-01-17 06:08:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:08:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:08:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 20, 243024), True, datetime.datetime(2023, 1, 17, 6, 8, 20, 243024), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:08:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:08:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:08:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:08:22,625 [243] [DEBUG] [app] Starting request: urn:request:5de4c091-cf8b-44b0-b2a3-d9f88be1857c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:22,626 [241] [DEBUG] [app] Starting request: urn:request:da71d88f-39aa-4bed-bef2-d500206f212f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:22,630 [242] [DEBUG] [app] Starting request: urn:request:c3cfcce1-2560-47de-9836-d651aa83cc0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:22,630 [240] [DEBUG] [app] Starting request: urn:request:79ad4cfa-6b4e-420e-83ea-edfe0cae6529 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:22,630 [242] [DEBUG] [app] Ending request: urn:request:c3cfcce1-2560-47de-9836-d651aa83cc0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3cfcce1-2560-47de-9836-d651aa83cc0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:08:22,631 [240] [DEBUG] [app] Ending request: urn:request:79ad4cfa-6b4e-420e-83ea-edfe0cae6529 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79ad4cfa-6b4e-420e-83ea-edfe0cae6529', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:08:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:22,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:22,635 [239] [DEBUG] [app] Starting request: urn:request:a4ef0864-fcca-49a5-a22a-c1c5044ef764 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:22,635 [239] [DEBUG] [app] Ending request: urn:request:a4ef0864-fcca-49a5-a22a-c1c5044ef764 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4ef0864-fcca-49a5-a22a-c1c5044ef764', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:22,635 [244] [DEBUG] [app] Starting request: urn:request:7a196149-19a3-4600-a737-bef7af0fd2e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:22,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:22,636 [244] [DEBUG] [app] Ending request: urn:request:7a196149-19a3-4600-a737-bef7af0fd2e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a196149-19a3-4600-a737-bef7af0fd2e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:08:22,636 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:22,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:22,647 [243] [DEBUG] [app] Ending request: urn:request:5de4c091-cf8b-44b0-b2a3-d9f88be1857c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5de4c091-cf8b-44b0-b2a3-d9f88be1857c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:22,647 [241] [DEBUG] [app] Ending request: urn:request:da71d88f-39aa-4bed-bef2-d500206f212f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:da71d88f-39aa-4bed-bef2-d500206f212f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:22,647 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) gunicorn-web stdout | 2023-01-17 06:08:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:08:22,647 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:08:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:39.230038+00:00 (in 16.392219 seconds) securityworker stdout | 2023-01-17 06:08:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:52 UTC)" (scheduled at 2023-01-17 06:08:22.837365+00:00) securityworker stdout | 2023-01-17 06:08:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:08:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:08:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:48.830274+00:00 (in 24.188054 seconds) exportactionlogsworker stdout | 2023-01-17 06:08:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:24 UTC)" (scheduled at 2023-01-17 06:08:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:08:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:08:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:08:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:08:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:20.242198+00:00 (in 54.907527 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:08:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:25 UTC)" (scheduled at 2023-01-17 06:08:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:08:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:08:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:25 UTC)" executed successfully builder stdout | 2023-01-17 06:08:25,872 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:25,881 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:25,881 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:25,887 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:08:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:08:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:57.226424+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 06:08:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:57 UTC)" (scheduled at 2023-01-17 06:08:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:08:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 8, 27, 227264), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:08:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:08:27,236 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:08:27,236 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:38.735066+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 06:08:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:38 UTC)" (scheduled at 2023-01-17 06:08:28.735066+00:00) notificationworker stdout | 2023-01-17 06:08:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 28, 735949), True, datetime.datetime(2023, 1, 17, 6, 8, 28, 735949), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:08:32,341 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:08:32,623 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:08:32,796 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:08:33,676 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:08:33,714 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:08:34,912 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:08:35,645 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:08:35,689 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:08:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:08:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:05.848675+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 06:08:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:05 UTC)" (scheduled at 2023-01-17 06:08:35.848675+00:00) gcworker stdout | 2023-01-17 06:08:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726115849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:08:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:08:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:08:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:05 UTC)" executed successfully builder stdout | 2023-01-17 06:08:35,887 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:35,895 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:35,895 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:35,901 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:08:36,137 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:08:36,459 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:08:36,461 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:08:36,464 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:08:37,019 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:08:37,160 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:08:37,625 [243] [DEBUG] [app] Starting request: urn:request:bfc79e89-4ba3-4b18-b409-fa63f5b453e8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:37,626 [244] [DEBUG] [app] Starting request: urn:request:8377ceb5-45b5-4ced-a7b0-804a50575d0e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:37,630 [240] [DEBUG] [app] Starting request: urn:request:80106521-3860-4f59-a80d-0e05a5710b4e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:37,631 [232] [DEBUG] [app] Starting request: urn:request:a397a079-8a10-431b-97a3-41feb4e1f1d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:37,631 [240] [DEBUG] [app] Ending request: urn:request:80106521-3860-4f59-a80d-0e05a5710b4e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:80106521-3860-4f59-a80d-0e05a5710b4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:08:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:08:37,631 [232] [DEBUG] [app] Ending request: urn:request:a397a079-8a10-431b-97a3-41feb4e1f1d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a397a079-8a10-431b-97a3-41feb4e1f1d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:08:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:37,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:37,636 [239] [DEBUG] [app] Starting request: urn:request:7760cd39-e0fc-4892-b014-dc53ec429870 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:37,636 [241] [DEBUG] [app] Starting request: urn:request:cd122137-c6a9-46ab-a884-922cfe4a132d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:37,636 [239] [DEBUG] [app] Ending request: urn:request:7760cd39-e0fc-4892-b014-dc53ec429870 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7760cd39-e0fc-4892-b014-dc53ec429870', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:37,637 [241] [DEBUG] [app] Ending request: urn:request:cd122137-c6a9-46ab-a884-922cfe4a132d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cd122137-c6a9-46ab-a884-922cfe4a132d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:08:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:37,648 [243] [DEBUG] [app] Ending request: urn:request:bfc79e89-4ba3-4b18-b409-fa63f5b453e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bfc79e89-4ba3-4b18-b409-fa63f5b453e8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:37,648 [244] [DEBUG] [app] Ending request: urn:request:8377ceb5-45b5-4ced-a7b0-804a50575d0e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8377ceb5-45b5-4ced-a7b0-804a50575d0e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:37,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:37,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:08:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:08:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:08:37,881 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:08:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:43.935629+00:00 (in 5.200087 seconds) notificationworker stdout | 2023-01-17 06:08:38,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:48 UTC)" (scheduled at 2023-01-17 06:08:38.735066+00:00) notificationworker stdout | 2023-01-17 06:08:38,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:38,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 38, 736441), True, datetime.datetime(2023, 1, 17, 6, 8, 38, 736441), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:48 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:08:38,765 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:08:38,767 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-registry stdout | 2023-01-17 06:08:38,934 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:08:38,937 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:08:38,940 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:08:38,942 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:08:38,945 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:08:38,947 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:08:38,950 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:08:38,952 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:08:38,954 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:08:38,962 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:08:38,966 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:08:38,968 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:08:38,973 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:08:38,978 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:08:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:08:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:48.738458+00:00 (in 9.707238 seconds) repositorygcworker stdout | 2023-01-17 06:08:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:39 UTC)" (scheduled at 2023-01-17 06:08:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:08:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:08:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 39, 31657), True, datetime.datetime(2023, 1, 17, 6, 8, 39, 31657), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:08:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:08:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:08:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:08:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:52.837365+00:00 (in 13.606881 seconds) securityworker stdout | 2023-01-17 06:08:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:09 UTC)" (scheduled at 2023-01-17 06:08:39.230038+00:00) securityworker stdout | 2023-01-17 06:08:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:08:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:08:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:08:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:08:39,877 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:08:40,511 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:08:41,342 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:08:41,379 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:08:41,434 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:08:41,530 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:08:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:48.735066+00:00 (in 4.798988 seconds) notificationworker stdout | 2023-01-17 06:08:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:43 UTC)" (scheduled at 2023-01-17 06:08:43.935629+00:00) notificationworker stdout | 2023-01-17 06:08:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:08:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:43 UTC)" executed successfully builder stdout | 2023-01-17 06:08:45,906 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:45,914 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:45,914 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:45,920 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:08:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:08:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:59.337413+00:00 (in 11.404426 seconds) namespacegcworker stdout | 2023-01-17 06:08:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:47 UTC)" (scheduled at 2023-01-17 06:08:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:08:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:08:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:56.935055+00:00 (in 8.199463 seconds) notificationworker stdout | 2023-01-17 06:08:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:58 UTC)" (scheduled at 2023-01-17 06:08:48.735066+00:00) notificationworker stdout | 2023-01-17 06:08:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 48, 736037), True, datetime.datetime(2023, 1, 17, 6, 8, 48, 736037), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:08:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:08:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:39.030814+00:00 (in 50.291869 seconds) repositorygcworker stdout | 2023-01-17 06:08:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:48 UTC)" (scheduled at 2023-01-17 06:08:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:08:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:08:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:08:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:08:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:24.641816+00:00 (in 35.811113 seconds) exportactionlogsworker stdout | 2023-01-17 06:08:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:48 UTC)" (scheduled at 2023-01-17 06:08:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:08:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:08:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 48, 831045), True, datetime.datetime(2023, 1, 17, 6, 8, 48, 831045), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:08:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:08:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:08:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:08:52,626 [241] [DEBUG] [app] Starting request: urn:request:c4232c55-46e8-4a87-ba4c-7d3bd8ea863c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,626 [239] [DEBUG] [app] Starting request: urn:request:a332a928-d5f1-4e09-93fa-e5a6b8ce53d1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:52,631 [236] [DEBUG] [app] Starting request: urn:request:582b4b5d-7cc4-4134-a970-e0513bc1456e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:52,631 [242] [DEBUG] [app] Starting request: urn:request:677e9e91-d211-427d-bc4e-6e629c0955a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:52,631 [236] [DEBUG] [app] Ending request: urn:request:582b4b5d-7cc4-4134-a970-e0513bc1456e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:582b4b5d-7cc4-4134-a970-e0513bc1456e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:08:52,631 [242] [DEBUG] [app] Ending request: urn:request:677e9e91-d211-427d-bc4e-6e629c0955a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:677e9e91-d211-427d-bc4e-6e629c0955a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:08:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:08:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:52,636 [244] [DEBUG] [app] Starting request: urn:request:48f30e9a-a477-4a6f-bb13-37446bec0f99 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,636 [243] [DEBUG] [app] Starting request: urn:request:11e957a5-0e9c-4ef2-a01d-61b9ab13b176 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,636 [244] [DEBUG] [app] Ending request: urn:request:48f30e9a-a477-4a6f-bb13-37446bec0f99 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48f30e9a-a477-4a6f-bb13-37446bec0f99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:08:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:52,636 [243] [DEBUG] [app] Ending request: urn:request:11e957a5-0e9c-4ef2-a01d-61b9ab13b176 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11e957a5-0e9c-4ef2-a01d-61b9ab13b176', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:52,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:52,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:52,647 [241] [DEBUG] [app] Ending request: urn:request:c4232c55-46e8-4a87-ba4c-7d3bd8ea863c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4232c55-46e8-4a87-ba4c-7d3bd8ea863c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:52,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:52,647 [239] [DEBUG] [app] Ending request: urn:request:a332a928-d5f1-4e09-93fa-e5a6b8ce53d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a332a928-d5f1-4e09-93fa-e5a6b8ce53d1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 06:08:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:08:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:09.230038+00:00 (in 16.392284 seconds) securityworker stdout | 2023-01-17 06:08:52,837 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:22 UTC)" (scheduled at 2023-01-17 06:08:52.837365+00:00) securityworker stdout | 2023-01-17 06:08:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:22 UTC)" executed successfully builder stdout | 2023-01-17 06:08:55,930 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:55,939 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:55,939 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:55,945 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:58.735066+00:00 (in 1.799517 seconds) notificationworker stdout | 2023-01-17 06:08:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:13:56 UTC)" (scheduled at 2023-01-17 06:08:56.935055+00:00) notificationworker stdout | 2023-01-17 06:08:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 8, 56, 935891), 'notification/%']) notificationworker stdout | 2023-01-17 06:08:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 8, 56, 935891), True, datetime.datetime(2023, 1, 17, 6, 8, 56, 935891), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:08:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 8, 56, 935891), True, datetime.datetime(2023, 1, 17, 6, 8, 56, 935891), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 8, 56, 935891), 'notification/%']) notificationworker stdout | 2023-01-17 06:08:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:13:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:08:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:08:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:27.226424+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 06:08:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:27 UTC)" (scheduled at 2023-01-17 06:08:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:08:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 8, 57, 227232), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:08:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:08:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:08:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:08.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 06:08:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:08 UTC)" (scheduled at 2023-01-17 06:08:58.735066+00:00) notificationworker stdout | 2023-01-17 06:08:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 58, 735987), True, datetime.datetime(2023, 1, 17, 6, 8, 58, 735987), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:08:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:08:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:15.931994+00:00 (in 16.594142 seconds) namespacegcworker stdout | 2023-01-17 06:08:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:59 UTC)" (scheduled at 2023-01-17 06:08:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:08:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:08:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 59, 338228), True, datetime.datetime(2023, 1, 17, 6, 8, 59, 338228), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:08:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:08:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:08:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:09:02,377 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:09:02,638 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:09:02,831 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:09:03,688 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:09:03,746 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:09:04,935 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:09:05,681 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:09:05,724 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:09:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:09:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:35.848675+00:00 (in 29.999510 seconds) gcworker stdout | 2023-01-17 06:09:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:35 UTC)" (scheduled at 2023-01-17 06:09:05.848675+00:00) gcworker stdout | 2023-01-17 06:09:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726145849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:09:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:09:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:09:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:35 UTC)" executed successfully builder stdout | 2023-01-17 06:09:05,955 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:05,963 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:05,963 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:05,969 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:09:06,163 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:09:06,465 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:09:06,468 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:09:06,471 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:09:07,054 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:09:07,195 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:09:07,625 [239] [DEBUG] [app] Starting request: urn:request:aaa1f504-cc0c-45f8-b707-3cec061c9033 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:07,625 [244] [DEBUG] [app] Starting request: urn:request:a15e85f2-e4f0-4af6-9fe8-a5bdc4c7a046 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:07,630 [240] [DEBUG] [app] Starting request: urn:request:522f2484-387a-437e-ac4d-3607ec3f1a34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,630 [242] [DEBUG] [app] Starting request: urn:request:6de5690c-fb5c-4f24-8569-fe498aad8793 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,631 [240] [DEBUG] [app] Ending request: urn:request:522f2484-387a-437e-ac4d-3607ec3f1a34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:522f2484-387a-437e-ac4d-3607ec3f1a34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,631 [242] [DEBUG] [app] Ending request: urn:request:6de5690c-fb5c-4f24-8569-fe498aad8793 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6de5690c-fb5c-4f24-8569-fe498aad8793', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:09:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:07,635 [244] [DEBUG] [app] Starting request: urn:request:96bc5c11-08c8-4c30-abeb-3eb1416922ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:07,636 [244] [DEBUG] [app] Ending request: urn:request:96bc5c11-08c8-4c30-abeb-3eb1416922ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:96bc5c11-08c8-4c30-abeb-3eb1416922ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:09:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:07,636 [243] [DEBUG] [app] Starting request: urn:request:de74f0da-2314-4157-8fb2-ca69fbed336b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,636 [243] [DEBUG] [app] Ending request: urn:request:de74f0da-2314-4157-8fb2-ca69fbed336b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de74f0da-2314-4157-8fb2-ca69fbed336b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:07,647 [244] [DEBUG] [app] Ending request: urn:request:a15e85f2-e4f0-4af6-9fe8-a5bdc4c7a046 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a15e85f2-e4f0-4af6-9fe8-a5bdc4c7a046', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:07,647 [239] [DEBUG] [app] Ending request: urn:request:aaa1f504-cc0c-45f8-b707-3cec061c9033 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aaa1f504-cc0c-45f8-b707-3cec061c9033', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:09:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:09:07,916 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:09:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:18.735066+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 06:09:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:18 UTC)" (scheduled at 2023-01-17 06:09:08.735066+00:00) notificationworker stdout | 2023-01-17 06:09:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 8, 735998), True, datetime.datetime(2023, 1, 17, 6, 9, 8, 735998), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:18 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:09:08,787 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:09:08,799 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:09:08,941 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:09:08,944 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:09:08,947 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:09:08,949 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:09:08,952 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:09:08,955 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:09:08,959 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:09:08,962 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:09:08,964 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:09:08,968 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:09:08,972 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:09:08,974 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:09:08,978 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:09:08,985 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:09:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:22.837365+00:00 (in 13.606956 seconds) securityworker stdout | 2023-01-17 06:09:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:39 UTC)" (scheduled at 2023-01-17 06:09:09.230038+00:00) securityworker stdout | 2023-01-17 06:09:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:09:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:09:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:09,233 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:09,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:09,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:09:09,240 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:09:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:09:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:09:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:09:09,909 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:09:10,547 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:09:11,377 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:09:11,410 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:09:11,466 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:09:11,564 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} namespacegcworker stdout | 2023-01-17 06:09:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:09:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:47.932490+00:00 (in 32.000013 seconds) namespacegcworker stdout | 2023-01-17 06:09:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:14:15 UTC)" (scheduled at 2023-01-17 06:09:15.931994+00:00) namespacegcworker stdout | 2023-01-17 06:09:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 9, 15, 932832), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:09:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 9, 15, 932832), True, datetime.datetime(2023, 1, 17, 6, 9, 15, 932832), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:09:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 9, 15, 932832), True, datetime.datetime(2023, 1, 17, 6, 9, 15, 932832), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 9, 15, 932832), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:09:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:09:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:14:15 UTC)" executed successfully builder stdout | 2023-01-17 06:09:15,974 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:15,982 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:15,982 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:15,987 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:28.735066+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 06:09:18,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:28 UTC)" (scheduled at 2023-01-17 06:09:18.735066+00:00) notificationworker stdout | 2023-01-17 06:09:18,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:18,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 18, 736169), True, datetime.datetime(2023, 1, 17, 6, 9, 18, 736169), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:09:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:09:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:25.334138+00:00 (in 5.091496 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:09:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:20 UTC)" (scheduled at 2023-01-17 06:09:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:09:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:09:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 20, 243053), True, datetime.datetime(2023, 1, 17, 6, 9, 20, 243053), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:09:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:09:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:09:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:09:22,625 [239] [DEBUG] [app] Starting request: urn:request:ef42b2ea-315b-4113-8bbe-50146f129f86 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:22,626 [244] [DEBUG] [app] Starting request: urn:request:3acad869-89f1-4de2-a627-019f56012b32 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:22,631 [232] [DEBUG] [app] Starting request: urn:request:3e1d50a6-f816-4d2e-9323-97be3b157057 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:22,631 [242] [DEBUG] [app] Starting request: urn:request:69501f97-a997-4860-b29b-376c854c8eb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:22,631 [232] [DEBUG] [app] Ending request: urn:request:3e1d50a6-f816-4d2e-9323-97be3b157057 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e1d50a6-f816-4d2e-9323-97be3b157057', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:09:22,632 [242] [DEBUG] [app] Ending request: urn:request:69501f97-a997-4860-b29b-376c854c8eb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69501f97-a997-4860-b29b-376c854c8eb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:09:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:22,636 [244] [DEBUG] [app] Starting request: urn:request:fbec8898-08b5-4fdb-a9e7-271801478856 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:22,637 [244] [DEBUG] [app] Ending request: urn:request:fbec8898-08b5-4fdb-a9e7-271801478856 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fbec8898-08b5-4fdb-a9e7-271801478856', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:22,637 [243] [DEBUG] [app] Starting request: urn:request:de915563-d134-41f2-a127-3abd4d9195a3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:22,637 [243] [DEBUG] [app] Ending request: urn:request:de915563-d134-41f2-a127-3abd4d9195a3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de915563-d134-41f2-a127-3abd4d9195a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:22,648 [244] [DEBUG] [app] Ending request: urn:request:3acad869-89f1-4de2-a627-019f56012b32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3acad869-89f1-4de2-a627-019f56012b32', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:09:22,648 [239] [DEBUG] [app] Ending request: urn:request:ef42b2ea-315b-4113-8bbe-50146f129f86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ef42b2ea-315b-4113-8bbe-50146f129f86', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) securityworker stdout | 2023-01-17 06:09:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:39.230038+00:00 (in 16.392242 seconds) securityworker stdout | 2023-01-17 06:09:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:52 UTC)" (scheduled at 2023-01-17 06:09:22.837365+00:00) securityworker stdout | 2023-01-17 06:09:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:09:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:09:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:48.830274+00:00 (in 24.188025 seconds) exportactionlogsworker stdout | 2023-01-17 06:09:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:24 UTC)" (scheduled at 2023-01-17 06:09:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:09:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:09:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:09:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:09:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:20.242198+00:00 (in 54.907540 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:09:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:25 UTC)" (scheduled at 2023-01-17 06:09:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:09:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:09:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:25 UTC)" executed successfully builder stdout | 2023-01-17 06:09:25,998 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:26,006 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:26,007 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:26,012 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:09:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:09:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:57.226424+00:00 (in 29.999540 seconds) buildlogsarchiver stdout | 2023-01-17 06:09:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:57 UTC)" (scheduled at 2023-01-17 06:09:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:09:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 9, 27, 227268), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:09:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:09:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:09:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:38.735066+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 06:09:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:38 UTC)" (scheduled at 2023-01-17 06:09:28.735066+00:00) notificationworker stdout | 2023-01-17 06:09:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 28, 735946), True, datetime.datetime(2023, 1, 17, 6, 9, 28, 735946), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:28,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:09:32,413 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:09:32,673 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:09:32,848 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:09:33,723 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:09:33,781 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:09:34,971 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:09:35,715 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:09:35,741 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:09:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:09:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:05.848675+00:00 (in 29.999545 seconds) gcworker stdout | 2023-01-17 06:09:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:05 UTC)" (scheduled at 2023-01-17 06:09:35.848675+00:00) gcworker stdout | 2023-01-17 06:09:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726175849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:09:35,860 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:09:35,860 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:09:35,860 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:05 UTC)" executed successfully builder stdout | 2023-01-17 06:09:36,022 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:36,030 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:36,030 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:36,036 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:09:36,199 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:09:36,472 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:09:36,475 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:09:36,477 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:09:37,090 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:09:37,206 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:09:37,625 [241] [DEBUG] [app] Starting request: urn:request:7ce6939d-61a0-4660-a88e-664a6544b899 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:37,625 [239] [DEBUG] [app] Starting request: urn:request:04d4cd0c-c3a4-4135-83cc-770ceaab6c75 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:37,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,628 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:37,629 [242] [DEBUG] [app] Starting request: urn:request:f5512462-007e-4abe-bfe8-3545e0279913 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,629 [242] [DEBUG] [app] Ending request: urn:request:f5512462-007e-4abe-bfe8-3545e0279913 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5512462-007e-4abe-bfe8-3545e0279913', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,629 [240] [DEBUG] [app] Starting request: urn:request:193003f9-bb26-488e-ad71-8ef072209d81 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:09:37,630 [240] [DEBUG] [app] Ending request: urn:request:193003f9-bb26-488e-ad71-8ef072209d81 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:193003f9-bb26-488e-ad71-8ef072209d81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:37,630 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:09:37,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:37,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:37,631 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,633 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:37,634 [239] [DEBUG] [app] Starting request: urn:request:9a2d6e64-5d58-4bba-922b-ad7612c6d837 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:37,634 [239] [DEBUG] [app] Ending request: urn:request:9a2d6e64-5d58-4bba-922b-ad7612c6d837 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9a2d6e64-5d58-4bba-922b-ad7612c6d837', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:37,635 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:37,635 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:37,635 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:37,636 [243] [DEBUG] [app] Starting request: urn:request:bc6b74d9-dfc4-422e-a32a-0a6596a04eea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:37,637 [243] [DEBUG] [app] Ending request: urn:request:bc6b74d9-dfc4-422e-a32a-0a6596a04eea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc6b74d9-dfc4-422e-a32a-0a6596a04eea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:09:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:37,640 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:37,645 [241] [DEBUG] [app] Ending request: urn:request:7ce6939d-61a0-4660-a88e-664a6544b899 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ce6939d-61a0-4660-a88e-664a6544b899', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:37,645 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:37,646 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.021 47 0.022) gunicorn-web stdout | 2023-01-17 06:09:37,647 [239] [DEBUG] [app] Ending request: urn:request:04d4cd0c-c3a4-4135-83cc-770ceaab6c75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04d4cd0c-c3a4-4135-83cc-770ceaab6c75', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:09:37,953 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:09:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:43.935629+00:00 (in 5.200105 seconds) notificationworker stdout | 2023-01-17 06:09:38,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:48 UTC)" (scheduled at 2023-01-17 06:09:38.735066+00:00) notificationworker stdout | 2023-01-17 06:09:38,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 38, 736135), True, datetime.datetime(2023, 1, 17, 6, 9, 38, 736135), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:48 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:09:38,815 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:09:38,834 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:09:38,949 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:09:38,952 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:09:38,954 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:09:38,957 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:09:38,960 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:09:38,962 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:09:38,965 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:09:38,972 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:09:38,974 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:09:38,977 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:09:38,979 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:09:38,982 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:09:38,984 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:09:38,991 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:09:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:09:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:48.738458+00:00 (in 9.707240 seconds) repositorygcworker stdout | 2023-01-17 06:09:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:39 UTC)" (scheduled at 2023-01-17 06:09:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:09:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:09:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 39, 31612), True, datetime.datetime(2023, 1, 17, 6, 9, 39, 31612), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:09:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:09:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:09:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:09:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:52.837365+00:00 (in 13.606860 seconds) securityworker stdout | 2023-01-17 06:09:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:09 UTC)" (scheduled at 2023-01-17 06:09:39.230038+00:00) securityworker stdout | 2023-01-17 06:09:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:09:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:09:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:09:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:09:39,937 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:09:40,582 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:09:41,412 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:09:41,444 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:09:41,494 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:09:41,579 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:09:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:48.735066+00:00 (in 4.799016 seconds) notificationworker stdout | 2023-01-17 06:09:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:43 UTC)" (scheduled at 2023-01-17 06:09:43.935629+00:00) notificationworker stdout | 2023-01-17 06:09:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:09:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:43 UTC)" executed successfully builder stdout | 2023-01-17 06:09:46,046 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:46,055 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:46,055 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:46,061 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:09:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:09:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:59.337413+00:00 (in 11.404432 seconds) namespacegcworker stdout | 2023-01-17 06:09:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:47 UTC)" (scheduled at 2023-01-17 06:09:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:09:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:09:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:48,736 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:58.735066+00:00 (in 9.999084 seconds) notificationworker stdout | 2023-01-17 06:09:48,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:58 UTC)" (scheduled at 2023-01-17 06:09:48.735066+00:00) notificationworker stdout | 2023-01-17 06:09:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:48,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 48, 736409), True, datetime.datetime(2023, 1, 17, 6, 9, 48, 736409), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:09:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:09:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:39.030814+00:00 (in 50.291880 seconds) repositorygcworker stdout | 2023-01-17 06:09:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:48 UTC)" (scheduled at 2023-01-17 06:09:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:09:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:09:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:09:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:09:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:24.641816+00:00 (in 35.811117 seconds) exportactionlogsworker stdout | 2023-01-17 06:09:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:48 UTC)" (scheduled at 2023-01-17 06:09:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:09:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:09:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 48, 831057), True, datetime.datetime(2023, 1, 17, 6, 9, 48, 831057), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:09:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:09:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:09:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:09:52,625 [244] [DEBUG] [app] Starting request: urn:request:6af58956-ba98-410c-b21e-9c7d5b547a66 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:52,625 [239] [DEBUG] [app] Starting request: urn:request:692e3936-b7c5-4301-96be-72035129e247 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:09:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:52,630 [242] [DEBUG] [app] Starting request: urn:request:056da095-b659-4b7e-9962-30d83e23255e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:52,630 [232] [DEBUG] [app] Starting request: urn:request:6c2656da-1ce7-4c9a-ac11-d44c5b7e5919 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:52,630 [242] [DEBUG] [app] Ending request: urn:request:056da095-b659-4b7e-9962-30d83e23255e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:056da095-b659-4b7e-9962-30d83e23255e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:09:52,630 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:09:52,630 [232] [DEBUG] [app] Ending request: urn:request:6c2656da-1ce7-4c9a-ac11-d44c5b7e5919 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c2656da-1ce7-4c9a-ac11-d44c5b7e5919', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:52,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:09:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:09:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:52,634 [243] [DEBUG] [app] Starting request: urn:request:730e14aa-ad9e-4f1f-b792-244120eb245c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:52,635 [244] [DEBUG] [app] Starting request: urn:request:13189f15-d998-46d6-b1db-1737ae8dbb2f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:52,635 [243] [DEBUG] [app] Ending request: urn:request:730e14aa-ad9e-4f1f-b792-244120eb245c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:730e14aa-ad9e-4f1f-b792-244120eb245c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:52,635 [244] [DEBUG] [app] Ending request: urn:request:13189f15-d998-46d6-b1db-1737ae8dbb2f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:13189f15-d998-46d6-b1db-1737ae8dbb2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:52,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:52,635 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:52,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:52,635 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:52,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:52,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:52,646 [244] [DEBUG] [app] Ending request: urn:request:6af58956-ba98-410c-b21e-9c7d5b547a66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6af58956-ba98-410c-b21e-9c7d5b547a66', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:52,646 [239] [DEBUG] [app] Ending request: urn:request:692e3936-b7c5-4301-96be-72035129e247 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:692e3936-b7c5-4301-96be-72035129e247', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:52,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:52,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:52,646 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:09:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) securityworker stdout | 2023-01-17 06:09:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:09.230038+00:00 (in 16.392186 seconds) securityworker stdout | 2023-01-17 06:09:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:22 UTC)" (scheduled at 2023-01-17 06:09:52.837365+00:00) securityworker stdout | 2023-01-17 06:09:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:22 UTC)" executed successfully builder stdout | 2023-01-17 06:09:56,069 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:56,078 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:56,078 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:56,084 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:09:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:09:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:27.226424+00:00 (in 29.999554 seconds) buildlogsarchiver stdout | 2023-01-17 06:09:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:27 UTC)" (scheduled at 2023-01-17 06:09:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:09:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 9, 57, 227220), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:09:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:09:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:09:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:08.735066+00:00 (in 9.999502 seconds) notificationworker stdout | 2023-01-17 06:09:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:08 UTC)" (scheduled at 2023-01-17 06:09:58.735066+00:00) notificationworker stdout | 2023-01-17 06:09:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 58, 735974), True, datetime.datetime(2023, 1, 17, 6, 9, 58, 735974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:09:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:09:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:47.932490+00:00 (in 48.594589 seconds) namespacegcworker stdout | 2023-01-17 06:09:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:59 UTC)" (scheduled at 2023-01-17 06:09:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:09:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:09:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 59, 338306), True, datetime.datetime(2023, 1, 17, 6, 9, 59, 338306), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:09:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:09:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:09:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:10:02,448 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:10:02,691 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:10:02,882 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:10:03,759 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:10:03,816 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:10:05,006 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} manifestbackfillworker stdout | 2023-01-17 06:10:05,752 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:10:05,776 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:10:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:10:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:35.848675+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 06:10:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:35 UTC)" (scheduled at 2023-01-17 06:10:05.848675+00:00) gcworker stdout | 2023-01-17 06:10:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726205849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:10:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:10:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:10:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:35 UTC)" executed successfully builder stdout | 2023-01-17 06:10:06,087 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:06,096 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:06,096 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:06,101 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:10:06,230 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:10:06,479 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:10:06,482 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:10:06,484 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:10:07,126 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:10:07,211 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:10:07,626 [239] [DEBUG] [app] Starting request: urn:request:0dc6d6a3-2e69-4380-b38d-dbf3d948ea52 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:07,626 [241] [DEBUG] [app] Starting request: urn:request:44257516-2935-4f33-ade6-2046b3e96197 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:07,631 [232] [DEBUG] [app] Starting request: urn:request:6e0da72c-8bfd-4c18-aa21-642db11735d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,631 [232] [DEBUG] [app] Ending request: urn:request:6e0da72c-8bfd-4c18-aa21-642db11735d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e0da72c-8bfd-4c18-aa21-642db11735d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,631 [240] [DEBUG] [app] Starting request: urn:request:803214dd-db76-4fe8-8b02-94d64fc667bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:10:07,632 [240] [DEBUG] [app] Ending request: urn:request:803214dd-db76-4fe8-8b02-94d64fc667bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:803214dd-db76-4fe8-8b02-94d64fc667bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:10:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:10:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:07,636 [244] [DEBUG] [app] Starting request: urn:request:02c79143-8607-419e-b17d-3c20421e77b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:07,636 [239] [DEBUG] [app] Starting request: urn:request:fb9405a0-6f1b-4f81-8a05-55f8130775ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:07,636 [244] [DEBUG] [app] Ending request: urn:request:02c79143-8607-419e-b17d-3c20421e77b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02c79143-8607-419e-b17d-3c20421e77b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:07,636 [239] [DEBUG] [app] Ending request: urn:request:fb9405a0-6f1b-4f81-8a05-55f8130775ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb9405a0-6f1b-4f81-8a05-55f8130775ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:07,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:07,644 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:07,647 [239] [DEBUG] [app] Ending request: urn:request:0dc6d6a3-2e69-4380-b38d-dbf3d948ea52 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0dc6d6a3-2e69-4380-b38d-dbf3d948ea52', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 06:10:07,650 [241] [DEBUG] [app] Ending request: urn:request:44257516-2935-4f33-ade6-2046b3e96197 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44257516-2935-4f33-ade6-2046b3e96197', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:07,650 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:07,650 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) chunkcleanupworker stdout | 2023-01-17 06:10:07,988 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:10:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:18.735066+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:10:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:18 UTC)" (scheduled at 2023-01-17 06:10:08.735066+00:00) notificationworker stdout | 2023-01-17 06:10:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 8, 735930), True, datetime.datetime(2023, 1, 17, 6, 10, 8, 735930), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:18 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:10:08,850 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:10:08,869 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:10:08,957 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:10:08,960 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:10:08,963 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:10:08,966 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:10:08,969 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:10:08,971 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:10:08,973 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:10:08,981 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:10:08,983 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:10:08,986 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-web stdout | 2023-01-17 06:10:08,989 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:10:08,991 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:10:08,993 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:10:08,996 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:10:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:22.837365+00:00 (in 13.606852 seconds) securityworker stdout | 2023-01-17 06:10:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:39 UTC)" (scheduled at 2023-01-17 06:10:09.230038+00:00) securityworker stdout | 2023-01-17 06:10:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:10:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:10:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:10:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:10:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:10:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:10:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:10:09,950 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:10:10,618 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:10:11,421 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:10:11,464 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:10:11,527 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:10:11,614 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:10:16,112 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:16,120 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:16,120 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:16,126 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:28.735066+00:00 (in 9.999472 seconds) notificationworker stdout | 2023-01-17 06:10:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:28 UTC)" (scheduled at 2023-01-17 06:10:18.735066+00:00) notificationworker stdout | 2023-01-17 06:10:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 18, 735963), True, datetime.datetime(2023, 1, 17, 6, 10, 18, 735963), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:10:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:10:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:25.334138+00:00 (in 5.091421 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:10:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:20 UTC)" (scheduled at 2023-01-17 06:10:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:10:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:10:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 20, 243144), True, datetime.datetime(2023, 1, 17, 6, 10, 20, 243144), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:10:20,253 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:10:20,253 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:10:20,254 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:10:22,625 [239] [DEBUG] [app] Starting request: urn:request:9e2a48c7-56c6-4726-b0ec-1e38803e1313 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:22,625 [243] [DEBUG] [app] Starting request: urn:request:b25db79c-3f34-40ce-ba54-f4cdf9120e8f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:22,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:22,630 [242] [DEBUG] [app] Starting request: urn:request:1527e0ce-e1d8-4904-8900-54ee16a491ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:22,630 [237] [DEBUG] [app] Starting request: urn:request:b32ab8ab-7c5b-4d0b-920b-58ec7233298f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:22,631 [242] [DEBUG] [app] Ending request: urn:request:1527e0ce-e1d8-4904-8900-54ee16a491ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1527e0ce-e1d8-4904-8900-54ee16a491ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:10:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:10:22,631 [237] [DEBUG] [app] Ending request: urn:request:b32ab8ab-7c5b-4d0b-920b-58ec7233298f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b32ab8ab-7c5b-4d0b-920b-58ec7233298f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:10:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:22,635 [244] [DEBUG] [app] Starting request: urn:request:01f5cdf6-3e7f-423a-bde6-82533133dd10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:22,636 [244] [DEBUG] [app] Ending request: urn:request:01f5cdf6-3e7f-423a-bde6-82533133dd10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01f5cdf6-3e7f-423a-bde6-82533133dd10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:22,636 [241] [DEBUG] [app] Starting request: urn:request:6955a540-0d1d-402a-9c27-b15c32577b02 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:10:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:22,636 [241] [DEBUG] [app] Ending request: urn:request:6955a540-0d1d-402a-9c27-b15c32577b02 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6955a540-0d1d-402a-9c27-b15c32577b02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:22,647 [239] [DEBUG] [app] Ending request: urn:request:9e2a48c7-56c6-4726-b0ec-1e38803e1313 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e2a48c7-56c6-4726-b0ec-1e38803e1313', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:22,647 [243] [DEBUG] [app] Ending request: urn:request:b25db79c-3f34-40ce-ba54-f4cdf9120e8f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b25db79c-3f34-40ce-ba54-f4cdf9120e8f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:10:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:10:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:39.230038+00:00 (in 16.392223 seconds) securityworker stdout | 2023-01-17 06:10:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:52 UTC)" (scheduled at 2023-01-17 06:10:22.837365+00:00) securityworker stdout | 2023-01-17 06:10:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:10:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:10:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:48.830274+00:00 (in 24.188025 seconds) exportactionlogsworker stdout | 2023-01-17 06:10:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:24 UTC)" (scheduled at 2023-01-17 06:10:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:10:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:10:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:10:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:10:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:20.242198+00:00 (in 54.907601 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:10:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:25 UTC)" (scheduled at 2023-01-17 06:10:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:10:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:10:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:25 UTC)" executed successfully builder stdout | 2023-01-17 06:10:26,133 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:26,142 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:26,142 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:26,147 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:10:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:10:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:57.226424+00:00 (in 29.999542 seconds) buildlogsarchiver stdout | 2023-01-17 06:10:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:57 UTC)" (scheduled at 2023-01-17 06:10:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:10:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 10, 27, 227234), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:10:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:10:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:10:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:38.735066+00:00 (in 9.999483 seconds) notificationworker stdout | 2023-01-17 06:10:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:38 UTC)" (scheduled at 2023-01-17 06:10:28.735066+00:00) notificationworker stdout | 2023-01-17 06:10:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 28, 735993), True, datetime.datetime(2023, 1, 17, 6, 10, 28, 735993), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:28,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:10:32,455 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:10:32,722 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:10:32,918 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:10:33,765 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:10:33,842 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:10:35,039 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:10:35,786 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:10:35,788 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gcworker stdout | 2023-01-17 06:10:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:10:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:05.848675+00:00 (in 29.999501 seconds) gcworker stdout | 2023-01-17 06:10:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:05 UTC)" (scheduled at 2023-01-17 06:10:35.848675+00:00) gcworker stdout | 2023-01-17 06:10:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726235849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:10:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:10:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:10:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:05 UTC)" executed successfully builder stdout | 2023-01-17 06:10:36,151 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:36,159 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:36,159 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:36,165 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:10:36,262 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:10:36,486 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:10:36,488 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:10:36,491 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:10:37,158 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:10:37,242 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:10:37,626 [239] [DEBUG] [app] Starting request: urn:request:b87b358e-88af-4ee5-bf1f-ffa84eb4d84f (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:37,626 [244] [DEBUG] [app] Starting request: urn:request:34c56eb9-b36b-441a-b035-6b63e55e1a04 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:37,631 [240] [DEBUG] [app] Starting request: urn:request:8a75b7a3-46be-48ab-8de1-93c7d5499a3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,631 [232] [DEBUG] [app] Starting request: urn:request:1acf1ab6-342c-4d0a-9223-6f5f544bfeb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,631 [240] [DEBUG] [app] Ending request: urn:request:8a75b7a3-46be-48ab-8de1-93c7d5499a3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a75b7a3-46be-48ab-8de1-93c7d5499a3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:10:37,632 [232] [DEBUG] [app] Ending request: urn:request:1acf1ab6-342c-4d0a-9223-6f5f544bfeb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1acf1ab6-342c-4d0a-9223-6f5f544bfeb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:10:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:10:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:37,636 [243] [DEBUG] [app] Starting request: urn:request:e34f8f6a-b474-44a2-8b94-82cf7ef52227 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:37,636 [244] [DEBUG] [app] Starting request: urn:request:8300b26c-78b2-45e7-9465-980710620836 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:37,636 [243] [DEBUG] [app] Ending request: urn:request:e34f8f6a-b474-44a2-8b94-82cf7ef52227 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e34f8f6a-b474-44a2-8b94-82cf7ef52227', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:37,637 [244] [DEBUG] [app] Ending request: urn:request:8300b26c-78b2-45e7-9465-980710620836 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8300b26c-78b2-45e7-9465-980710620836', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:10:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:37,648 [244] [DEBUG] [app] Ending request: urn:request:34c56eb9-b36b-441a-b035-6b63e55e1a04 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:34c56eb9-b36b-441a-b035-6b63e55e1a04', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:37,648 [239] [DEBUG] [app] Ending request: urn:request:b87b358e-88af-4ee5-bf1f-ffa84eb4d84f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b87b358e-88af-4ee5-bf1f-ffa84eb4d84f', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:10:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:10:38,022 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:10:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:43.935629+00:00 (in 5.200088 seconds) notificationworker stdout | 2023-01-17 06:10:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:48 UTC)" (scheduled at 2023-01-17 06:10:38.735066+00:00) notificationworker stdout | 2023-01-17 06:10:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 38, 735906), True, datetime.datetime(2023, 1, 17, 6, 10, 38, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:48 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:10:38,886 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:10:38,888 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:10:38,964 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:10:38,971 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:10:38,974 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:10:38,977 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:10:38,979 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:10:38,981 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:10:38,984 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:10:38,989 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:10:38,992 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:10:38,994 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:10:38,997 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:10:38,999 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:10:39,002 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:10:39,004 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:10:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:10:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:48.738458+00:00 (in 9.707248 seconds) repositorygcworker stdout | 2023-01-17 06:10:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:39 UTC)" (scheduled at 2023-01-17 06:10:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:10:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:10:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 39, 31615), True, datetime.datetime(2023, 1, 17, 6, 10, 39, 31615), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:10:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:10:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:10:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:10:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:52.837365+00:00 (in 13.606868 seconds) securityworker stdout | 2023-01-17 06:10:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:09 UTC)" (scheduled at 2023-01-17 06:10:39.230038+00:00) securityworker stdout | 2023-01-17 06:10:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:10:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:10:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:39,253 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:39,253 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:10:39,253 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:10:39,253 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:10:39,253 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:39,253 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:39,254 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:39,254 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:10:39,254 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:10:39,983 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:10:40,642 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:10:41,428 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:10:41,475 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:10:41,531 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:10:41,621 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:10:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:48.735066+00:00 (in 4.798974 seconds) notificationworker stdout | 2023-01-17 06:10:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:43 UTC)" (scheduled at 2023-01-17 06:10:43.935629+00:00) notificationworker stdout | 2023-01-17 06:10:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:10:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:43 UTC)" executed successfully builder stdout | 2023-01-17 06:10:46,165 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:46,173 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:46,174 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:46,179 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:10:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:10:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:59.337413+00:00 (in 11.404476 seconds) namespacegcworker stdout | 2023-01-17 06:10:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:47 UTC)" (scheduled at 2023-01-17 06:10:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:10:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:10:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:58.735066+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 06:10:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:58 UTC)" (scheduled at 2023-01-17 06:10:48.735066+00:00) notificationworker stdout | 2023-01-17 06:10:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 48, 735954), True, datetime.datetime(2023, 1, 17, 6, 10, 48, 735954), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:10:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:10:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:39.030814+00:00 (in 50.291915 seconds) repositorygcworker stdout | 2023-01-17 06:10:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:48 UTC)" (scheduled at 2023-01-17 06:10:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:10:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:10:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:10:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:10:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:24.641816+00:00 (in 35.811094 seconds) exportactionlogsworker stdout | 2023-01-17 06:10:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:48 UTC)" (scheduled at 2023-01-17 06:10:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:10:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:10:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 48, 831094), True, datetime.datetime(2023, 1, 17, 6, 10, 48, 831094), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:10:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:10:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:10:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:48 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 06:10:50,627 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 06:10:50,628 [72] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 07:10:50.626417+00:00 (in 3599.998309 seconds) manifestbackfillworker stdout | 2023-01-17 06:10:50,628 [72] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2023-01-17 07:10:50 UTC)" (scheduled at 2023-01-17 06:10:50.626417+00:00) manifestbackfillworker stdout | 2023-01-17 06:10:50,628 [72] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0]) manifestbackfillworker stdout | 2023-01-17 06:10:50,635 [72] [DEBUG] [__main__] Manifest backfill worker has completed; skipping manifestbackfillworker stdout | 2023-01-17 06:10:50,635 [72] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:10:50,635 [72] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2023-01-17 07:10:50 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:10:52,625 [243] [DEBUG] [app] Starting request: urn:request:5878b5d4-b11f-48d2-bb5c-29bc883aebc7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:52,625 [239] [DEBUG] [app] Starting request: urn:request:195888b8-8924-4fe2-ba1c-c3b966ab780d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:10:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:52,630 [242] [DEBUG] [app] Starting request: urn:request:2f780748-f858-4b56-8259-95f35422df2c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:52,630 [240] [DEBUG] [app] Starting request: urn:request:5c18dcd0-71ea-4e40-a286-d452095da5af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:52,631 [242] [DEBUG] [app] Ending request: urn:request:2f780748-f858-4b56-8259-95f35422df2c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f780748-f858-4b56-8259-95f35422df2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:10:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:10:52,631 [240] [DEBUG] [app] Ending request: urn:request:5c18dcd0-71ea-4e40-a286-d452095da5af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c18dcd0-71ea-4e40-a286-d452095da5af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:10:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:52,636 [244] [DEBUG] [app] Starting request: urn:request:f5f664b8-be1e-4eaa-b097-fd036509bedc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:52,636 [241] [DEBUG] [app] Starting request: urn:request:89de6cc8-1923-4b67-8a6a-91a2329376a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:52,636 [244] [DEBUG] [app] Ending request: urn:request:f5f664b8-be1e-4eaa-b097-fd036509bedc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f5f664b8-be1e-4eaa-b097-fd036509bedc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:52,636 [241] [DEBUG] [app] Ending request: urn:request:89de6cc8-1923-4b67-8a6a-91a2329376a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89de6cc8-1923-4b67-8a6a-91a2329376a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:52,648 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:52,652 [243] [DEBUG] [app] Ending request: urn:request:5878b5d4-b11f-48d2-bb5c-29bc883aebc7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5878b5d4-b11f-48d2-bb5c-29bc883aebc7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:52,652 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:52,652 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:10:52,654 [239] [DEBUG] [app] Ending request: urn:request:195888b8-8924-4fe2-ba1c-c3b966ab780d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:195888b8-8924-4fe2-ba1c-c3b966ab780d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:52,654 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:52,654 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityworker stdout | 2023-01-17 06:10:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:09.230038+00:00 (in 16.392250 seconds) securityworker stdout | 2023-01-17 06:10:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:22 UTC)" (scheduled at 2023-01-17 06:10:52.837365+00:00) securityworker stdout | 2023-01-17 06:10:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:22 UTC)" executed successfully builder stdout | 2023-01-17 06:10:56,189 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:56,198 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:56,198 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:56,204 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:10:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:10:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:27.226424+00:00 (in 29.999578 seconds) buildlogsarchiver stdout | 2023-01-17 06:10:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:27 UTC)" (scheduled at 2023-01-17 06:10:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:10:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 10, 57, 227201), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:10:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:10:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:10:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:08.735066+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 06:10:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:08 UTC)" (scheduled at 2023-01-17 06:10:58.735066+00:00) notificationworker stdout | 2023-01-17 06:10:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 58, 735974), True, datetime.datetime(2023, 1, 17, 6, 10, 58, 735974), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:58,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:10:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:10:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:47.932490+00:00 (in 48.594606 seconds) namespacegcworker stdout | 2023-01-17 06:10:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:59 UTC)" (scheduled at 2023-01-17 06:10:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:10:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:10:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 59, 338422), True, datetime.datetime(2023, 1, 17, 6, 10, 59, 338422), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:10:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:10:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:10:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:11:02,480 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:11:02,757 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:11:02,954 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:11:03,789 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:11:03,877 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:11:05,074 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:11:05,801 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:11:05,823 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gcworker stdout | 2023-01-17 06:11:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:11:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:35.848675+00:00 (in 29.999529 seconds) gcworker stdout | 2023-01-17 06:11:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:35 UTC)" (scheduled at 2023-01-17 06:11:05.848675+00:00) gcworker stdout | 2023-01-17 06:11:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726265849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:11:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:11:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:11:05,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:35 UTC)" executed successfully builder stdout | 2023-01-17 06:11:06,206 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:06,214 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:06,215 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:06,220 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:11:06,296 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:11:06,493 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:11:06,495 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:11:06,498 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:11:07,165 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:11:07,264 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:11:07,625 [244] [DEBUG] [app] Starting request: urn:request:4e4e0653-ce1e-42b1-8d7b-ee0346615220 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:07,626 [243] [DEBUG] [app] Starting request: urn:request:94edeb90-39b5-4bb3-b2be-7fb30b49c4fb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:07,630 [240] [DEBUG] [app] Starting request: urn:request:0c7d9941-7326-46d6-992a-67fef6bc760a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:07,631 [232] [DEBUG] [app] Starting request: urn:request:1c208357-8cfc-420e-8a55-00302698393e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:07,631 [240] [DEBUG] [app] Ending request: urn:request:0c7d9941-7326-46d6-992a-67fef6bc760a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c7d9941-7326-46d6-992a-67fef6bc760a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:11:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:11:07,631 [232] [DEBUG] [app] Ending request: urn:request:1c208357-8cfc-420e-8a55-00302698393e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c208357-8cfc-420e-8a55-00302698393e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:11:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:11:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:07,635 [239] [DEBUG] [app] Starting request: urn:request:827eaf80-eac8-420e-a7c8-04284f65d6d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:07,636 [241] [DEBUG] [app] Starting request: urn:request:f8207422-e211-48d4-94ab-884cd929b0d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:07,636 [239] [DEBUG] [app] Ending request: urn:request:827eaf80-eac8-420e-a7c8-04284f65d6d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:827eaf80-eac8-420e-a7c8-04284f65d6d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:11:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:07,636 [241] [DEBUG] [app] Ending request: urn:request:f8207422-e211-48d4-94ab-884cd929b0d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8207422-e211-48d4-94ab-884cd929b0d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:07,647 [243] [DEBUG] [app] Ending request: urn:request:94edeb90-39b5-4bb3-b2be-7fb30b49c4fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94edeb90-39b5-4bb3-b2be-7fb30b49c4fb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:07,647 [244] [DEBUG] [app] Ending request: urn:request:4e4e0653-ce1e-42b1-8d7b-ee0346615220 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4e4e0653-ce1e-42b1-8d7b-ee0346615220', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:11:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:11:08,058 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:11:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:18.735066+00:00 (in 9.999537 seconds) notificationworker stdout | 2023-01-17 06:11:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:18 UTC)" (scheduled at 2023-01-17 06:11:08.735066+00:00) notificationworker stdout | 2023-01-17 06:11:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 8, 735901), True, datetime.datetime(2023, 1, 17, 6, 11, 8, 735901), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:18 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:11:08,922 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:11:08,924 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:11:08,971 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:11:08,982 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:11:08,985 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:11:08,987 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:11:08,990 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:11:08,993 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:11:08,996 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:11:08,999 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:11:09,001 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:11:09,004 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:11:09,006 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:11:09,008 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:11:09,011 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:11:09,013 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:11:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:22.837365+00:00 (in 13.606873 seconds) securityworker stdout | 2023-01-17 06:11:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:39 UTC)" (scheduled at 2023-01-17 06:11:09.230038+00:00) securityworker stdout | 2023-01-17 06:11:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:11:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:11:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:11:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:11:09,993 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:11:10,678 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:11:11,438 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:11:11,500 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:11:11,540 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:11:11,656 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:11:16,230 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:16,239 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:16,239 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:16,245 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:28.735066+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 06:11:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:28 UTC)" (scheduled at 2023-01-17 06:11:18.735066+00:00) notificationworker stdout | 2023-01-17 06:11:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 18, 735879), True, datetime.datetime(2023, 1, 17, 6, 11, 18, 735879), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:11:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:11:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:25.334138+00:00 (in 5.091489 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:11:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:20 UTC)" (scheduled at 2023-01-17 06:11:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:11:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:11:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 20, 243056), True, datetime.datetime(2023, 1, 17, 6, 11, 20, 243056), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:11:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:11:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:11:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:11:22,625 [241] [DEBUG] [app] Starting request: urn:request:a96fd9e0-7ac0-49bb-83a9-9eecf0d11475 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:22,625 [244] [DEBUG] [app] Starting request: urn:request:2de3a83e-c01d-42b2-ae10-86164569c2fb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:22,630 [240] [DEBUG] [app] Starting request: urn:request:89082828-5482-4bc4-ad50-3e4942f6607d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:22,631 [240] [DEBUG] [app] Ending request: urn:request:89082828-5482-4bc4-ad50-3e4942f6607d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:89082828-5482-4bc4-ad50-3e4942f6607d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:22,631 [237] [DEBUG] [app] Starting request: urn:request:1aa87cc9-f560-4734-9fcc-e234dcabc61b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:11:22,631 [237] [DEBUG] [app] Ending request: urn:request:1aa87cc9-f560-4734-9fcc-e234dcabc61b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1aa87cc9-f560-4734-9fcc-e234dcabc61b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:11:22,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:22,635 [239] [DEBUG] [app] Starting request: urn:request:7d764cfc-bf7b-442a-9bc2-1b5a1efdb9b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:22,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:22,636 [239] [DEBUG] [app] Ending request: urn:request:7d764cfc-bf7b-442a-9bc2-1b5a1efdb9b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d764cfc-bf7b-442a-9bc2-1b5a1efdb9b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:22,636 [244] [DEBUG] [app] Starting request: urn:request:0f605e7d-8532-4441-a69a-703f572b679b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:11:22,637 [244] [DEBUG] [app] Ending request: urn:request:0f605e7d-8532-4441-a69a-703f572b679b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0f605e7d-8532-4441-a69a-703f572b679b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:11:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:22,648 [244] [DEBUG] [app] Ending request: urn:request:2de3a83e-c01d-42b2-ae10-86164569c2fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2de3a83e-c01d-42b2-ae10-86164569c2fb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:22,648 [241] [DEBUG] [app] Ending request: urn:request:a96fd9e0-7ac0-49bb-83a9-9eecf0d11475 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a96fd9e0-7ac0-49bb-83a9-9eecf0d11475', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:22,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:11:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:11:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:22,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:39.230038+00:00 (in 16.392194 seconds) securityworker stdout | 2023-01-17 06:11:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:52 UTC)" (scheduled at 2023-01-17 06:11:22.837365+00:00) securityworker stdout | 2023-01-17 06:11:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:11:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:11:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:44.641497+00:00 (in 19.999236 seconds) exportactionlogsworker stdout | 2023-01-17 06:11:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:24 UTC)" (scheduled at 2023-01-17 06:11:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:11:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:11:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:11:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:11:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:39.333860+00:00 (in 13.999248 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:11:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:25 UTC)" (scheduled at 2023-01-17 06:11:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:11:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:11:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:25 UTC)" executed successfully builder stdout | 2023-01-17 06:11:26,255 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:26,264 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:26,264 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:26,269 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:11:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:11:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:57.226424+00:00 (in 29.999551 seconds) buildlogsarchiver stdout | 2023-01-17 06:11:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:57 UTC)" (scheduled at 2023-01-17 06:11:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:11:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 11, 27, 227223), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:11:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:11:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:11:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:38.735066+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 06:11:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:38 UTC)" (scheduled at 2023-01-17 06:11:28.735066+00:00) notificationworker stdout | 2023-01-17 06:11:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 28, 735954), True, datetime.datetime(2023, 1, 17, 6, 11, 28, 735954), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:11:32,516 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:11:32,778 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:11:32,990 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:11:33,824 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:11:33,912 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:11:35,110 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} builder stdout | 2023-01-17 06:11:35,829 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} gcworker stdout | 2023-01-17 06:11:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:11:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:05.848675+00:00 (in 29.999549 seconds) gcworker stdout | 2023-01-17 06:11:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:05 UTC)" (scheduled at 2023-01-17 06:11:35.848675+00:00) gcworker stdout | 2023-01-17 06:11:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726295849, 1209600, True, 3, 500, 1, 0]) manifestbackfillworker stdout | 2023-01-17 06:11:35,857 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gcworker stdout | 2023-01-17 06:11:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:11:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:11:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:05 UTC)" executed successfully builder stdout | 2023-01-17 06:11:36,272 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:36,280 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:36,280 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:36,286 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:11:36,301 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:11:36,501 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:11:36,503 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:11:36,505 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:11:37,201 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:11:37,299 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:11:37,625 [239] [DEBUG] [app] Starting request: urn:request:44abe1ef-a924-43f6-8c7a-c2f87336eb18 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:37,626 [243] [DEBUG] [app] Starting request: urn:request:4ed84967-faf0-4ebe-864c-4d0e4cd61dbb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:37,630 [240] [DEBUG] [app] Starting request: urn:request:8392bfed-fa26-4ca0-8f0c-24a3b9c7ec60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,631 [242] [DEBUG] [app] Starting request: urn:request:40f1eed7-e6f5-45d0-b5aa-7182a2208e06 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,631 [240] [DEBUG] [app] Ending request: urn:request:8392bfed-fa26-4ca0-8f0c-24a3b9c7ec60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8392bfed-fa26-4ca0-8f0c-24a3b9c7ec60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:11:37,631 [242] [DEBUG] [app] Ending request: urn:request:40f1eed7-e6f5-45d0-b5aa-7182a2208e06 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40f1eed7-e6f5-45d0-b5aa-7182a2208e06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:11:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:11:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:37,635 [244] [DEBUG] [app] Starting request: urn:request:ff39bc01-1820-45ef-b47b-7d24cccb49d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:37,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:37,636 [244] [DEBUG] [app] Ending request: urn:request:ff39bc01-1820-45ef-b47b-7d24cccb49d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff39bc01-1820-45ef-b47b-7d24cccb49d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:37,636 [239] [DEBUG] [app] Starting request: urn:request:bf647e5f-68c4-4334-92a3-c3c3737ad553 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:37,636 [239] [DEBUG] [app] Ending request: urn:request:bf647e5f-68c4-4334-92a3-c3c3737ad553 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf647e5f-68c4-4334-92a3-c3c3737ad553', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:37,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:37,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:37,647 [243] [DEBUG] [app] Ending request: urn:request:4ed84967-faf0-4ebe-864c-4d0e4cd61dbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ed84967-faf0-4ebe-864c-4d0e4cd61dbb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:37,648 [239] [DEBUG] [app] Ending request: urn:request:44abe1ef-a924-43f6-8c7a-c2f87336eb18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44abe1ef-a924-43f6-8c7a-c2f87336eb18', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:37,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:11:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:11:38,072 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:11:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:43.935629+00:00 (in 5.200077 seconds) notificationworker stdout | 2023-01-17 06:11:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:48 UTC)" (scheduled at 2023-01-17 06:11:38.735066+00:00) notificationworker stdout | 2023-01-17 06:11:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 38, 735968), True, datetime.datetime(2023, 1, 17, 6, 11, 38, 735968), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:48 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:11:38,950 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:11:38,956 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:11:38,981 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:11:38,992 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:11:38,994 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:11:38,996 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:11:38,999 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:11:39,001 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:11:39,005 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:11:39,008 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:11:39,010 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:11:39,013 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:11:39,015 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:11:39,017 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:11:39,020 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:11:39,023 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} repositorygcworker stdout | 2023-01-17 06:11:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:11:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:48.738458+00:00 (in 9.707252 seconds) repositorygcworker stdout | 2023-01-17 06:11:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:39 UTC)" (scheduled at 2023-01-17 06:11:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:11:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:11:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 39, 31580), True, datetime.datetime(2023, 1, 17, 6, 11, 39, 31580), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:11:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:11:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:11:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:39 UTC)" executed successfully securityworker stdout | 2023-01-17 06:11:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:52.837365+00:00 (in 13.606871 seconds) securityworker stdout | 2023-01-17 06:11:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:09 UTC)" (scheduled at 2023-01-17 06:11:39.230038+00:00) securityworker stdout | 2023-01-17 06:11:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:11:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:11:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:11:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:11:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:11:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:20.242198+00:00 (in 40.907866 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:11:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:39 UTC)" (scheduled at 2023-01-17 06:11:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:11:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 11, 39, 334764), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:11:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 39, 334764), True, datetime.datetime(2023, 1, 17, 6, 11, 39, 334764), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:11:39,344 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 39, 334764), True, datetime.datetime(2023, 1, 17, 6, 11, 39, 334764), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 11, 39, 334764), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:11:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:11:39,346 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:11:40,020 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:11:40,714 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:11:41,461 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:11:41,505 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:11:41,575 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:11:41,685 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:11:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:48.735066+00:00 (in 4.798917 seconds) notificationworker stdout | 2023-01-17 06:11:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:43 UTC)" (scheduled at 2023-01-17 06:11:43.935629+00:00) notificationworker stdout | 2023-01-17 06:11:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:11:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:11:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:11:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:48.830274+00:00 (in 4.188288 seconds) exportactionlogsworker stdout | 2023-01-17 06:11:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:44 UTC)" (scheduled at 2023-01-17 06:11:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 06:11:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 11, 44, 642376), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:11:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 44, 642376), True, datetime.datetime(2023, 1, 17, 6, 11, 44, 642376), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:11:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 44, 642376), True, datetime.datetime(2023, 1, 17, 6, 11, 44, 642376), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 11, 44, 642376), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:11:44,654 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:11:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:44 UTC)" executed successfully builder stdout | 2023-01-17 06:11:46,290 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:46,299 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:46,299 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:46,305 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:11:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:11:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:59.337413+00:00 (in 11.404368 seconds) namespacegcworker stdout | 2023-01-17 06:11:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:47 UTC)" (scheduled at 2023-01-17 06:11:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:11:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:11:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:58.735066+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 06:11:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:58 UTC)" (scheduled at 2023-01-17 06:11:48.735066+00:00) notificationworker stdout | 2023-01-17 06:11:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 48, 735948), True, datetime.datetime(2023, 1, 17, 6, 11, 48, 735948), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:11:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:11:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:04.738062+00:00 (in 15.999162 seconds) repositorygcworker stdout | 2023-01-17 06:11:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:48 UTC)" (scheduled at 2023-01-17 06:11:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:11:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:11:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:11:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:11:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:24.641816+00:00 (in 35.811136 seconds) exportactionlogsworker stdout | 2023-01-17 06:11:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:48 UTC)" (scheduled at 2023-01-17 06:11:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:11:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:11:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 48, 831037), True, datetime.datetime(2023, 1, 17, 6, 11, 48, 831037), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:11:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:11:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:11:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:11:52,625 [239] [DEBUG] [app] Starting request: urn:request:bf1b0c5a-8a5b-4393-9904-78bbc1018cc4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:52,625 [243] [DEBUG] [app] Starting request: urn:request:e4c5f0da-eb08-463d-ba11-044705c66ae5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:11:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:52,630 [232] [DEBUG] [app] Starting request: urn:request:0ff7e80b-4764-47c5-bdef-ed90eb3097e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:52,630 [232] [DEBUG] [app] Ending request: urn:request:0ff7e80b-4764-47c5-bdef-ed90eb3097e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ff7e80b-4764-47c5-bdef-ed90eb3097e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:52,630 [235] [DEBUG] [app] Starting request: urn:request:44e6bd1a-008a-45bd-bbe9-3754345998df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:11:52,631 [235] [DEBUG] [app] Ending request: urn:request:44e6bd1a-008a-45bd-bbe9-3754345998df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44e6bd1a-008a-45bd-bbe9-3754345998df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:11:52,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:11:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:52,635 [239] [DEBUG] [app] Starting request: urn:request:dd67d3a9-ca24-42c7-a12f-99f82c7f7529 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:52,636 [239] [DEBUG] [app] Ending request: urn:request:dd67d3a9-ca24-42c7-a12f-99f82c7f7529 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd67d3a9-ca24-42c7-a12f-99f82c7f7529', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:52,636 [244] [DEBUG] [app] Starting request: urn:request:e746a7df-4da5-4306-90f7-d8e874fecd83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:52,636 [244] [DEBUG] [app] Ending request: urn:request:e746a7df-4da5-4306-90f7-d8e874fecd83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e746a7df-4da5-4306-90f7-d8e874fecd83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:52,648 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:52,648 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:52,653 [239] [DEBUG] [app] Ending request: urn:request:bf1b0c5a-8a5b-4393-9904-78bbc1018cc4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf1b0c5a-8a5b-4393-9904-78bbc1018cc4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:52,653 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:52,653 [243] [DEBUG] [app] Ending request: urn:request:e4c5f0da-eb08-463d-ba11-044705c66ae5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4c5f0da-eb08-463d-ba11-044705c66ae5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:52,654 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:52,654 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 06:11:52,654 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:11:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:09.230038+00:00 (in 16.392274 seconds) securityworker stdout | 2023-01-17 06:11:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:22 UTC)" (scheduled at 2023-01-17 06:11:52.837365+00:00) securityworker stdout | 2023-01-17 06:11:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:22 UTC)" executed successfully builder stdout | 2023-01-17 06:11:56,315 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:56,323 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:56,323 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:56,329 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:11:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:11:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:27.226424+00:00 (in 29.999524 seconds) buildlogsarchiver stdout | 2023-01-17 06:11:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:27 UTC)" (scheduled at 2023-01-17 06:11:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:11:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 11, 57, 227252), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:11:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:11:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:11:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:08.735066+00:00 (in 9.999487 seconds) notificationworker stdout | 2023-01-17 06:11:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:08 UTC)" (scheduled at 2023-01-17 06:11:58.735066+00:00) notificationworker stdout | 2023-01-17 06:11:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 58, 736006), True, datetime.datetime(2023, 1, 17, 6, 11, 58, 736006), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:11:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:11:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:47.932490+00:00 (in 48.594523 seconds) namespacegcworker stdout | 2023-01-17 06:11:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:59 UTC)" (scheduled at 2023-01-17 06:11:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:11:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:11:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 59, 338400), True, datetime.datetime(2023, 1, 17, 6, 11, 59, 338400), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:11:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:11:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:11:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:12:02,528 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:12:02,808 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:12:03,025 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:12:03,831 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:12:03,918 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:12:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:12:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:39.030814+00:00 (in 34.292345 seconds) repositorygcworker stdout | 2023-01-17 06:12:04,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:17:04 UTC)" (scheduled at 2023-01-17 06:12:04.738062+00:00) repositorygcworker stdout | 2023-01-17 06:12:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 12, 4, 739124), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:12:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 12, 4, 739124), True, datetime.datetime(2023, 1, 17, 6, 12, 4, 739124), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:12:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 12, 4, 739124), True, datetime.datetime(2023, 1, 17, 6, 12, 4, 739124), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 12, 4, 739124), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:12:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:12:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:17:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:12:05,131 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:12:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:12:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:35.848675+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 06:12:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:35 UTC)" (scheduled at 2023-01-17 06:12:05.848675+00:00) gcworker stdout | 2023-01-17 06:12:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726325849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:12:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:12:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:12:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:35 UTC)" executed successfully builder stdout | 2023-01-17 06:12:05,864 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:12:05,886 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:12:06,333 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue namespacegcworker stdout | 2023-01-17 06:12:06,337 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:12:06,342 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:06,342 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:06,348 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 06:12:06,507 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:12:06,510 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:12:06,512 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:12:07,237 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:12:07,320 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:12:07,625 [244] [DEBUG] [app] Starting request: urn:request:21e6aa63-1c4d-4071-ae3f-5226f1d2422a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:07,625 [243] [DEBUG] [app] Starting request: urn:request:79b9b424-438b-498f-8506-1f1d8980571e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:07,630 [242] [DEBUG] [app] Starting request: urn:request:ae95930a-4bcf-4ac8-8fbe-02690f5642bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,630 [240] [DEBUG] [app] Starting request: urn:request:ffb7759b-f9e9-450e-9d9d-9186ac3975f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,631 [242] [DEBUG] [app] Ending request: urn:request:ae95930a-4bcf-4ac8-8fbe-02690f5642bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ae95930a-4bcf-4ac8-8fbe-02690f5642bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,631 [240] [DEBUG] [app] Ending request: urn:request:ffb7759b-f9e9-450e-9d9d-9186ac3975f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ffb7759b-f9e9-450e-9d9d-9186ac3975f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:12:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:12:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:07,635 [239] [DEBUG] [app] Starting request: urn:request:4dcb43df-9463-41b9-9da8-56f8a09845df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:07,635 [241] [DEBUG] [app] Starting request: urn:request:10e96ac7-60df-4b64-ac1d-225d7eb2c3cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:07,635 [239] [DEBUG] [app] Ending request: urn:request:4dcb43df-9463-41b9-9da8-56f8a09845df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4dcb43df-9463-41b9-9da8-56f8a09845df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:07,636 [241] [DEBUG] [app] Ending request: urn:request:10e96ac7-60df-4b64-ac1d-225d7eb2c3cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10e96ac7-60df-4b64-ac1d-225d7eb2c3cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:12:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:07,647 [243] [DEBUG] [app] Ending request: urn:request:79b9b424-438b-498f-8506-1f1d8980571e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79b9b424-438b-498f-8506-1f1d8980571e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:07,647 [244] [DEBUG] [app] Ending request: urn:request:21e6aa63-1c4d-4071-ae3f-5226f1d2422a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21e6aa63-1c4d-4071-ae3f-5226f1d2422a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:12:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:12:08,085 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:12:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:18.735066+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 06:12:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:18 UTC)" (scheduled at 2023-01-17 06:12:08.735066+00:00) notificationworker stdout | 2023-01-17 06:12:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 8, 735900), True, datetime.datetime(2023, 1, 17, 6, 12, 8, 735900), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:18 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:12:08,987 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:12:08,989 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-registry stdout | 2023-01-17 06:12:08,992 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:12:09,001 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:12:09,003 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:12:09,006 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:12:09,009 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:12:09,011 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:12:09,014 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:12:09,016 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:12:09,019 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:12:09,021 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:12:09,024 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:12:09,027 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:12:09,029 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:12:09,031 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:12:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:22.837365+00:00 (in 13.606884 seconds) securityworker stdout | 2023-01-17 06:12:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:39 UTC)" (scheduled at 2023-01-17 06:12:09.230038+00:00) securityworker stdout | 2023-01-17 06:12:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:12:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:12:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:12:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:12:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:12:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:12:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:12:10,056 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:12:10,720 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:12:11,497 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:12:11,539 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:12:11,610 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:12:11,721 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:12:16,358 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:16,366 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:16,366 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:16,372 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:28.735066+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 06:12:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:28 UTC)" (scheduled at 2023-01-17 06:12:18.735066+00:00) notificationworker stdout | 2023-01-17 06:12:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 18, 735876), True, datetime.datetime(2023, 1, 17, 6, 12, 18, 735876), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:12:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:12:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:25.334138+00:00 (in 5.091343 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:12:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:20 UTC)" (scheduled at 2023-01-17 06:12:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:12:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:12:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 20, 243169), True, datetime.datetime(2023, 1, 17, 6, 12, 20, 243169), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:12:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:12:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:12:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:22,625 [239] [DEBUG] [app] Starting request: urn:request:c79ca720-6608-4697-90f2-15b57306c679 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:22,625 [241] [DEBUG] [app] Starting request: urn:request:f1a3e961-ea01-43e5-a438-23ef9a965545 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:22,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:22,630 [242] [DEBUG] [app] Starting request: urn:request:e837c103-2dda-4cda-8633-fed5fae4f985 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:22,631 [242] [DEBUG] [app] Ending request: urn:request:e837c103-2dda-4cda-8633-fed5fae4f985 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e837c103-2dda-4cda-8633-fed5fae4f985', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:22,631 [232] [DEBUG] [app] Starting request: urn:request:a3a69e52-f8e6-4453-87cd-113de6e4e3f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:12:22,631 [232] [DEBUG] [app] Ending request: urn:request:a3a69e52-f8e6-4453-87cd-113de6e4e3f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3a69e52-f8e6-4453-87cd-113de6e4e3f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:12:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:12:22,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:22,636 [244] [DEBUG] [app] Starting request: urn:request:6fdb0bd8-a823-40d4-872c-041c4a40e237 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:22,636 [239] [DEBUG] [app] Starting request: urn:request:7cc1186f-a493-4453-9064-b4d5755be093 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:22,636 [244] [DEBUG] [app] Ending request: urn:request:6fdb0bd8-a823-40d4-872c-041c4a40e237 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6fdb0bd8-a823-40d4-872c-041c4a40e237', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:22,636 [239] [DEBUG] [app] Ending request: urn:request:7cc1186f-a493-4453-9064-b4d5755be093 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7cc1186f-a493-4453-9064-b4d5755be093', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:22,648 [239] [DEBUG] [app] Ending request: urn:request:c79ca720-6608-4697-90f2-15b57306c679 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c79ca720-6608-4697-90f2-15b57306c679', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:22,648 [241] [DEBUG] [app] Ending request: urn:request:f1a3e961-ea01-43e5-a438-23ef9a965545 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1a3e961-ea01-43e5-a438-23ef9a965545', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:22,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:12:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:12:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:12:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:39.230038+00:00 (in 16.392244 seconds) securityworker stdout | 2023-01-17 06:12:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:52 UTC)" (scheduled at 2023-01-17 06:12:22.837365+00:00) securityworker stdout | 2023-01-17 06:12:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:12:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:12:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:48.830274+00:00 (in 24.188025 seconds) exportactionlogsworker stdout | 2023-01-17 06:12:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:24 UTC)" (scheduled at 2023-01-17 06:12:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:12:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:12:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:12:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:12:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:20.242198+00:00 (in 54.907644 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:12:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:25 UTC)" (scheduled at 2023-01-17 06:12:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:12:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:12:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:25 UTC)" executed successfully builder stdout | 2023-01-17 06:12:26,382 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:26,391 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:26,391 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:26,397 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:12:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:12:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:57.226424+00:00 (in 29.999549 seconds) buildlogsarchiver stdout | 2023-01-17 06:12:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:57 UTC)" (scheduled at 2023-01-17 06:12:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:12:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 12, 27, 227292), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:12:27,236 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:12:27,236 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:12:27,236 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:38.735066+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 06:12:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:38 UTC)" (scheduled at 2023-01-17 06:12:28.735066+00:00) notificationworker stdout | 2023-01-17 06:12:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 28, 735906), True, datetime.datetime(2023, 1, 17, 6, 12, 28, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:12:32,559 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:12:32,843 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:12:33,055 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:12:33,866 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:12:33,953 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:12:35,166 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:12:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:12:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:05.848675+00:00 (in 29.999541 seconds) gcworker stdout | 2023-01-17 06:12:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:05 UTC)" (scheduled at 2023-01-17 06:12:35.848675+00:00) gcworker stdout | 2023-01-17 06:12:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726355849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:12:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:12:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:12:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:05 UTC)" executed successfully builder stdout | 2023-01-17 06:12:35,900 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:12:35,920 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:12:36,350 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:12:36,407 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:36,416 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:36,416 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:36,421 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 06:12:36,513 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:12:36,517 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:12:36,519 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:12:37,273 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:12:37,342 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:12:37,626 [239] [DEBUG] [app] Starting request: urn:request:fded2cba-4795-49a4-a618-8935b3966ec6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:37,626 [244] [DEBUG] [app] Starting request: urn:request:b6a6ace5-3825-4fd6-8f79-0e68fb61db7c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:37,631 [242] [DEBUG] [app] Starting request: urn:request:97883da2-6aa9-4d87-b289-ef3043728714 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:37,631 [242] [DEBUG] [app] Ending request: urn:request:97883da2-6aa9-4d87-b289-ef3043728714 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97883da2-6aa9-4d87-b289-ef3043728714', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:37,631 [234] [DEBUG] [app] Starting request: urn:request:70f5b996-ca09-4242-9a4a-0e8638a9b0ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:12:37,632 [234] [DEBUG] [app] Ending request: urn:request:70f5b996-ca09-4242-9a4a-0e8638a9b0ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:70f5b996-ca09-4242-9a4a-0e8638a9b0ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:12:37,632 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:12:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:37,636 [244] [DEBUG] [app] Starting request: urn:request:f92c64ec-7a9d-443a-ab0f-f5ce6c9a6deb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:37,637 [244] [DEBUG] [app] Ending request: urn:request:f92c64ec-7a9d-443a-ab0f-f5ce6c9a6deb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f92c64ec-7a9d-443a-ab0f-f5ce6c9a6deb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:37,637 [241] [DEBUG] [app] Starting request: urn:request:820ac14b-850e-4a7e-a0f3-138f2006dfb9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:12:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:37,637 [241] [DEBUG] [app] Ending request: urn:request:820ac14b-850e-4a7e-a0f3-138f2006dfb9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:820ac14b-850e-4a7e-a0f3-138f2006dfb9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:37,648 [244] [DEBUG] [app] Ending request: urn:request:b6a6ace5-3825-4fd6-8f79-0e68fb61db7c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6a6ace5-3825-4fd6-8f79-0e68fb61db7c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:37,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:12:37,648 [239] [DEBUG] [app] Ending request: urn:request:fded2cba-4795-49a4-a618-8935b3966ec6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fded2cba-4795-49a4-a618-8935b3966ec6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:12:38,122 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:12:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:43.935629+00:00 (in 5.200119 seconds) notificationworker stdout | 2023-01-17 06:12:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:48 UTC)" (scheduled at 2023-01-17 06:12:38.735066+00:00) notificationworker stdout | 2023-01-17 06:12:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 38, 735906), True, datetime.datetime(2023, 1, 17, 6, 12, 38, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:12:39,003 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:12:39,006 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:12:39,011 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:12:39,013 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:12:39,018 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:12:39,021 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:12:39,024 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} logrotateworker stdout | 2023-01-17 06:12:39,026 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-registry stdout | 2023-01-17 06:12:39,029 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} repositorygcworker stdout | 2023-01-17 06:12:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run queuecleanupworker stdout | 2023-01-17 06:12:39,031 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} repositorygcworker stdout | 2023-01-17 06:12:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:48.738458+00:00 (in 9.707233 seconds) repositorygcworker stdout | 2023-01-17 06:12:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:39 UTC)" (scheduled at 2023-01-17 06:12:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:12:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:12:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 39, 31705), True, datetime.datetime(2023, 1, 17, 6, 12, 39, 31705), 0, 'repositorygc/%', 50, 1, 0]) gunicorn-web stdout | 2023-01-17 06:12:39,034 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:12:39,037 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} repositorygcworker stdout | 2023-01-17 06:12:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:12:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:12:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:39 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:39,039 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:12:39,043 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:12:39,045 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:12:39,048 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:12:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:52.837365+00:00 (in 13.606902 seconds) securityworker stdout | 2023-01-17 06:12:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:09 UTC)" (scheduled at 2023-01-17 06:12:39.230038+00:00) securityworker stdout | 2023-01-17 06:12:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:12:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:12:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:39,232 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:12:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:12:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:12:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:12:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:12:40,085 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:12:40,756 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:12:41,533 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:12:41,572 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:12:41,642 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:12:41,735 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:12:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:48.735066+00:00 (in 4.798897 seconds) notificationworker stdout | 2023-01-17 06:12:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:43 UTC)" (scheduled at 2023-01-17 06:12:43.935629+00:00) notificationworker stdout | 2023-01-17 06:12:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:12:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:43 UTC)" executed successfully builder stdout | 2023-01-17 06:12:46,432 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:46,441 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:46,441 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:46,447 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:12:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:12:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:59.337413+00:00 (in 11.404436 seconds) namespacegcworker stdout | 2023-01-17 06:12:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:47 UTC)" (scheduled at 2023-01-17 06:12:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:12:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:12:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:58.735066+00:00 (in 9.999423 seconds) notificationworker stdout | 2023-01-17 06:12:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:58 UTC)" (scheduled at 2023-01-17 06:12:48.735066+00:00) notificationworker stdout | 2023-01-17 06:12:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 48, 735998), True, datetime.datetime(2023, 1, 17, 6, 12, 48, 735998), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:12:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:12:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:39.030814+00:00 (in 50.291842 seconds) repositorygcworker stdout | 2023-01-17 06:12:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:48 UTC)" (scheduled at 2023-01-17 06:12:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:12:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:12:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:12:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:12:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:24.641816+00:00 (in 35.811090 seconds) exportactionlogsworker stdout | 2023-01-17 06:12:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:48 UTC)" (scheduled at 2023-01-17 06:12:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:12:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:12:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 48, 831066), True, datetime.datetime(2023, 1, 17, 6, 12, 48, 831066), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:12:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:12:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:12:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:52,626 [239] [DEBUG] [app] Starting request: urn:request:e3885444-e486-41bf-8091-4fb476bbdfcb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:52,626 [244] [DEBUG] [app] Starting request: urn:request:a7740c67-6b86-43a0-ae24-49539a6b57a8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:12:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:52,631 [242] [DEBUG] [app] Starting request: urn:request:27dd3ac2-a95e-41f6-bbdc-52342410dcc7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,631 [232] [DEBUG] [app] Starting request: urn:request:86595b3d-cdac-406d-a255-c7b0a6ad76e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,631 [242] [DEBUG] [app] Ending request: urn:request:27dd3ac2-a95e-41f6-bbdc-52342410dcc7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27dd3ac2-a95e-41f6-bbdc-52342410dcc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:12:52,631 [232] [DEBUG] [app] Ending request: urn:request:86595b3d-cdac-406d-a255-c7b0a6ad76e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86595b3d-cdac-406d-a255-c7b0a6ad76e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:12:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:52,636 [241] [DEBUG] [app] Starting request: urn:request:36e94204-223c-425a-9349-a923fd0d34d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:52,636 [239] [DEBUG] [app] Starting request: urn:request:49f2dafb-88b9-4246-8f2e-1bf68a9e4c52 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:52,637 [241] [DEBUG] [app] Ending request: urn:request:36e94204-223c-425a-9349-a923fd0d34d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36e94204-223c-425a-9349-a923fd0d34d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:52,637 [239] [DEBUG] [app] Ending request: urn:request:49f2dafb-88b9-4246-8f2e-1bf68a9e4c52 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49f2dafb-88b9-4246-8f2e-1bf68a9e4c52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:52,648 [244] [DEBUG] [app] Ending request: urn:request:a7740c67-6b86-43a0-ae24-49539a6b57a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7740c67-6b86-43a0-ae24-49539a6b57a8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:52,648 [239] [DEBUG] [app] Ending request: urn:request:e3885444-e486-41bf-8091-4fb476bbdfcb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3885444-e486-41bf-8091-4fb476bbdfcb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:12:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:12:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:09.230038+00:00 (in 16.392090 seconds) securityworker stdout | 2023-01-17 06:12:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:22 UTC)" (scheduled at 2023-01-17 06:12:52.837365+00:00) securityworker stdout | 2023-01-17 06:12:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:22 UTC)" executed successfully builder stdout | 2023-01-17 06:12:56,457 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:56,466 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:56,466 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:56,471 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:12:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:12:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:27.226424+00:00 (in 29.999506 seconds) buildlogsarchiver stdout | 2023-01-17 06:12:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:27 UTC)" (scheduled at 2023-01-17 06:12:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:12:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 12, 57, 227288), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:12:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:12:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:12:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:08.735066+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 06:12:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:08 UTC)" (scheduled at 2023-01-17 06:12:58.735066+00:00) notificationworker stdout | 2023-01-17 06:12:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 58, 735973), True, datetime.datetime(2023, 1, 17, 6, 12, 58, 735973), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:12:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:12:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:47.932490+00:00 (in 48.594542 seconds) namespacegcworker stdout | 2023-01-17 06:12:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:59 UTC)" (scheduled at 2023-01-17 06:12:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:12:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:12:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 59, 338445), True, datetime.datetime(2023, 1, 17, 6, 12, 59, 338445), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:12:59,345 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:12:59,345 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:12:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:13:02,577 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:13:02,875 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:13:03,068 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:13:03,893 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:13:03,989 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:13:05,202 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:13:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:13:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:35.848675+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 06:13:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:35 UTC)" (scheduled at 2023-01-17 06:13:05.848675+00:00) gcworker stdout | 2023-01-17 06:13:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726385849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:13:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:13:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:13:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:35 UTC)" executed successfully builder stdout | 2023-01-17 06:13:05,923 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:13:05,954 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:13:06,356 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:13:06,482 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:06,490 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:06,490 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:06,496 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 06:13:06,519 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:13:06,522 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:13:06,525 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} repositoryactioncounter stdout | 2023-01-17 06:13:07,309 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:13:07,377 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:13:07,626 [243] [DEBUG] [app] Starting request: urn:request:b20a5976-3ebf-41c7-9ee2-287471c395b0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,626 [239] [DEBUG] [app] Starting request: urn:request:f32e0d0e-0581-4a87-9d2f-14409c74158a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:07,630 [232] [DEBUG] [app] Starting request: urn:request:c7b00b73-f18e-4b39-b509-5342aaae9e98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:07,631 [232] [DEBUG] [app] Ending request: urn:request:c7b00b73-f18e-4b39-b509-5342aaae9e98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c7b00b73-f18e-4b39-b509-5342aaae9e98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:07,631 [242] [DEBUG] [app] Starting request: urn:request:c274b343-6332-4256-8982-ba73f2fa273e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:13:07,631 [242] [DEBUG] [app] Ending request: urn:request:c274b343-6332-4256-8982-ba73f2fa273e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c274b343-6332-4256-8982-ba73f2fa273e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:13:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:07,635 [243] [DEBUG] [app] Starting request: urn:request:e7548ff5-30b4-4069-a4a5-ee72b614f6d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,635 [243] [DEBUG] [app] Ending request: urn:request:e7548ff5-30b4-4069-a4a5-ee72b614f6d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7548ff5-30b4-4069-a4a5-ee72b614f6d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:07,637 [244] [DEBUG] [app] Starting request: urn:request:2f356e63-badf-417c-84db-c8c9c25b80ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,637 [244] [DEBUG] [app] Ending request: urn:request:2f356e63-badf-417c-84db-c8c9c25b80ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f356e63-badf-417c-84db-c8c9c25b80ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:07,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:07,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:07,639 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:07,646 [239] [DEBUG] [app] Ending request: urn:request:f32e0d0e-0581-4a87-9d2f-14409c74158a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f32e0d0e-0581-4a87-9d2f-14409c74158a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:07,646 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.021 47 0.021) gunicorn-web stdout | 2023-01-17 06:13:07,648 [243] [DEBUG] [app] Ending request: urn:request:b20a5976-3ebf-41c7-9ee2-287471c395b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b20a5976-3ebf-41c7-9ee2-287471c395b0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:13:08,142 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:13:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:18.735066+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 06:13:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:18 UTC)" (scheduled at 2023-01-17 06:13:08.735066+00:00) notificationworker stdout | 2023-01-17 06:13:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 8, 735877), True, datetime.datetime(2023, 1, 17, 6, 13, 8, 735877), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:13:09,013 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:13:09,015 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:13:09,018 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:13:09,021 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:13:09,025 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:13:09,027 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:13:09,030 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:13:09,035 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:13:09,040 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:13:09,042 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:13:09,047 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:13:09,051 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:13:09,053 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:13:09,056 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:13:09,061 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:13:09,065 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} securityworker stdout | 2023-01-17 06:13:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:22.837365+00:00 (in 13.606859 seconds) securityworker stdout | 2023-01-17 06:13:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:39 UTC)" (scheduled at 2023-01-17 06:13:09.230038+00:00) securityworker stdout | 2023-01-17 06:13:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:13:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:13:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:13:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:13:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:13:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:13:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:13:10,111 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:13:10,787 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:13:11,568 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:13:11,588 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:13:11,674 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:13:11,743 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:13:16,506 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:16,515 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:16,515 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:16,521 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:28.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 06:13:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:28 UTC)" (scheduled at 2023-01-17 06:13:18.735066+00:00) notificationworker stdout | 2023-01-17 06:13:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 18, 735941), True, datetime.datetime(2023, 1, 17, 6, 13, 18, 735941), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:13:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:13:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:25.334138+00:00 (in 5.091364 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:13:20,243 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:20 UTC)" (scheduled at 2023-01-17 06:13:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:13:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:13:20,244 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 20, 243257), True, datetime.datetime(2023, 1, 17, 6, 13, 20, 243257), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:13:20,252 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:13:20,252 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:13:20,252 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:13:22,625 [241] [DEBUG] [app] Starting request: urn:request:75fb4538-da47-4df4-9c6c-a7d5ba4509eb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:22,625 [239] [DEBUG] [app] Starting request: urn:request:4835be08-5735-460e-b051-850151b6e389 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:22,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:22,630 [237] [DEBUG] [app] Starting request: urn:request:cfca3dc5-6631-4bba-aa4f-cd52cf49df71 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:22,631 [237] [DEBUG] [app] Ending request: urn:request:cfca3dc5-6631-4bba-aa4f-cd52cf49df71 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cfca3dc5-6631-4bba-aa4f-cd52cf49df71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:22,631 [242] [DEBUG] [app] Starting request: urn:request:8174d1ef-fe5b-4a4f-9cae-ba774e37c29b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:13:22,631 [242] [DEBUG] [app] Ending request: urn:request:8174d1ef-fe5b-4a4f-9cae-ba774e37c29b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8174d1ef-fe5b-4a4f-9cae-ba774e37c29b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:22,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:13:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:22,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:22,636 [244] [DEBUG] [app] Starting request: urn:request:a5e154f9-948c-4d53-988c-b5bb474f25c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:22,636 [241] [DEBUG] [app] Starting request: urn:request:ed43b362-c673-4919-97c4-861fa1148a4b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:22,636 [244] [DEBUG] [app] Ending request: urn:request:a5e154f9-948c-4d53-988c-b5bb474f25c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a5e154f9-948c-4d53-988c-b5bb474f25c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:22,636 [241] [DEBUG] [app] Ending request: urn:request:ed43b362-c673-4919-97c4-861fa1148a4b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed43b362-c673-4919-97c4-861fa1148a4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:22,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:22,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:22,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:22,648 [241] [DEBUG] [app] Ending request: urn:request:75fb4538-da47-4df4-9c6c-a7d5ba4509eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75fb4538-da47-4df4-9c6c-a7d5ba4509eb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:22,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:22,648 [239] [DEBUG] [app] Ending request: urn:request:4835be08-5735-460e-b051-850151b6e389 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4835be08-5735-460e-b051-850151b6e389', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:22,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:13:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:13:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:39.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 06:13:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:52 UTC)" (scheduled at 2023-01-17 06:13:22.837365+00:00) securityworker stdout | 2023-01-17 06:13:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:22,851 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:22,852 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:22,852 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:13:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:13:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:48.830274+00:00 (in 24.188053 seconds) exportactionlogsworker stdout | 2023-01-17 06:13:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:24 UTC)" (scheduled at 2023-01-17 06:13:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:13:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:13:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:13:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:13:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:20.242198+00:00 (in 54.907508 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:13:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:25 UTC)" (scheduled at 2023-01-17 06:13:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:13:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:13:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:25 UTC)" executed successfully builder stdout | 2023-01-17 06:13:26,531 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:26,540 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:26,541 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:26,546 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:13:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:13:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:57.226424+00:00 (in 29.999560 seconds) buildlogsarchiver stdout | 2023-01-17 06:13:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:57 UTC)" (scheduled at 2023-01-17 06:13:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:13:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 13, 27, 227224), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:13:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:13:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:13:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:13:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:38.735066+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 06:13:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:38 UTC)" (scheduled at 2023-01-17 06:13:28.735066+00:00) notificationworker stdout | 2023-01-17 06:13:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 28, 735951), True, datetime.datetime(2023, 1, 17, 6, 13, 28, 735951), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:13:32,584 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:13:32,900 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:13:33,082 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:13:33,901 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:13:34,023 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:13:35,238 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:13:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:13:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:05.848675+00:00 (in 29.999545 seconds) gcworker stdout | 2023-01-17 06:13:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:05 UTC)" (scheduled at 2023-01-17 06:13:35.848675+00:00) gcworker stdout | 2023-01-17 06:13:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726415849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:13:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:13:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:13:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:05 UTC)" executed successfully builder stdout | 2023-01-17 06:13:35,944 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:13:35,980 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:13:36,366 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:13:36,526 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:13:36,529 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:13:36,531 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:13:36,551 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:36,560 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:36,560 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:36,566 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:13:37,343 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:13:37,412 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:13:37,625 [243] [DEBUG] [app] Starting request: urn:request:c152e08a-8974-467e-9a1e-c221e5ede48b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:37,626 [239] [DEBUG] [app] Starting request: urn:request:9055aedd-c0a3-475a-a212-d71722d2b9f9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:37,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:37,631 [232] [DEBUG] [app] Starting request: urn:request:60764991-634d-45c7-8082-3934612d2408 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:37,632 [242] [DEBUG] [app] Starting request: urn:request:f849e4b9-6df3-4ab9-bcef-a9933780d413 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:37,632 [232] [DEBUG] [app] Ending request: urn:request:60764991-634d-45c7-8082-3934612d2408 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60764991-634d-45c7-8082-3934612d2408', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:37,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:13:37,632 [242] [DEBUG] [app] Ending request: urn:request:f849e4b9-6df3-4ab9-bcef-a9933780d413 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f849e4b9-6df3-4ab9-bcef-a9933780d413', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:13:37,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:13:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:37,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:37,637 [244] [DEBUG] [app] Starting request: urn:request:78435fe7-f5f6-4b06-a098-9d41a54e5506 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:37,637 [244] [DEBUG] [app] Ending request: urn:request:78435fe7-f5f6-4b06-a098-9d41a54e5506 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78435fe7-f5f6-4b06-a098-9d41a54e5506', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:37,637 [241] [DEBUG] [app] Starting request: urn:request:d5bcf35d-b9ef-4577-8b1a-06baac602a1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:37,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:37,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:37,638 [241] [DEBUG] [app] Ending request: urn:request:d5bcf35d-b9ef-4577-8b1a-06baac602a1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5bcf35d-b9ef-4577-8b1a-06baac602a1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:37,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:37,639 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:37,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:37,649 [243] [DEBUG] [app] Ending request: urn:request:c152e08a-8974-467e-9a1e-c221e5ede48b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c152e08a-8974-467e-9a1e-c221e5ede48b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:37,649 [239] [DEBUG] [app] Ending request: urn:request:9055aedd-c0a3-475a-a212-d71722d2b9f9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9055aedd-c0a3-475a-a212-d71722d2b9f9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:37,649 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:37,649 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 06:13:37,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:13:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:13:38,177 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:13:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:43.935629+00:00 (in 5.200108 seconds) notificationworker stdout | 2023-01-17 06:13:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:48 UTC)" (scheduled at 2023-01-17 06:13:38.735066+00:00) notificationworker stdout | 2023-01-17 06:13:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 38, 735941), True, datetime.datetime(2023, 1, 17, 6, 13, 38, 735941), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:48 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:13:39,023 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:13:39,026 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:13:39,028 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:13:39,030 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} repositorygcworker stdout | 2023-01-17 06:13:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:13:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:48.738458+00:00 (in 9.707195 seconds) repositorygcworker stdout | 2023-01-17 06:13:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:39 UTC)" (scheduled at 2023-01-17 06:13:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:13:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:13:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 39, 31761), True, datetime.datetime(2023, 1, 17, 6, 13, 39, 31761), 0, 'repositorygc/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 06:13:39,033 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:13:39,035 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:13:39,037 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} repositorygcworker stdout | 2023-01-17 06:13:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:13:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:13:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:13:39,044 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:13:39,046 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:13:39,053 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:13:39,055 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:13:39,058 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:13:39,060 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:13:39,062 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:13:39,075 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:13:39,096 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} securityworker stdout | 2023-01-17 06:13:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:52.837365+00:00 (in 13.606914 seconds) securityworker stdout | 2023-01-17 06:13:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:09 UTC)" (scheduled at 2023-01-17 06:13:39.230038+00:00) securityworker stdout | 2023-01-17 06:13:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:13:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:13:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:13:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:13:40,147 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:13:40,823 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:13:41,598 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:13:41,601 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:13:41,691 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:13:41,749 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:13:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:48.735066+00:00 (in 4.798960 seconds) notificationworker stdout | 2023-01-17 06:13:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:43 UTC)" (scheduled at 2023-01-17 06:13:43.935629+00:00) notificationworker stdout | 2023-01-17 06:13:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:13:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:43 UTC)" executed successfully builder stdout | 2023-01-17 06:13:46,576 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:46,585 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:46,586 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:46,591 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:13:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:13:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:59.337413+00:00 (in 11.404322 seconds) namespacegcworker stdout | 2023-01-17 06:13:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:47 UTC)" (scheduled at 2023-01-17 06:13:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:13:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:13:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:13:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:56.935055+00:00 (in 8.199489 seconds) notificationworker stdout | 2023-01-17 06:13:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:58 UTC)" (scheduled at 2023-01-17 06:13:48.735066+00:00) notificationworker stdout | 2023-01-17 06:13:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 48, 735964), True, datetime.datetime(2023, 1, 17, 6, 13, 48, 735964), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:13:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:13:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:39.030814+00:00 (in 50.291843 seconds) repositorygcworker stdout | 2023-01-17 06:13:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:48 UTC)" (scheduled at 2023-01-17 06:13:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:13:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:13:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:13:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:13:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:13:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:24.641816+00:00 (in 35.811096 seconds) exportactionlogsworker stdout | 2023-01-17 06:13:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:48 UTC)" (scheduled at 2023-01-17 06:13:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:13:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:13:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 48, 831124), True, datetime.datetime(2023, 1, 17, 6, 13, 48, 831124), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:13:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:13:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:13:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:13:52,625 [241] [DEBUG] [app] Starting request: urn:request:3469f1fb-ed89-4b4b-ae51-0835972b66ea (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:52,625 [239] [DEBUG] [app] Starting request: urn:request:fa6589ed-907b-4578-94cc-cd92be2b77c5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:13:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:52,631 [236] [DEBUG] [app] Starting request: urn:request:d73eec58-4e37-46d1-a5e1-579182820e47 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,631 [237] [DEBUG] [app] Starting request: urn:request:007db33f-18ce-48bc-8f9c-3514700c9d67 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,631 [236] [DEBUG] [app] Ending request: urn:request:d73eec58-4e37-46d1-a5e1-579182820e47 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d73eec58-4e37-46d1-a5e1-579182820e47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:13:52,631 [237] [DEBUG] [app] Ending request: urn:request:007db33f-18ce-48bc-8f9c-3514700c9d67 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:007db33f-18ce-48bc-8f9c-3514700c9d67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:13:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:13:52,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:13:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:52,636 [244] [DEBUG] [app] Starting request: urn:request:ddda494e-c9e1-4de0-8b53-6322547d1d38 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:52,636 [243] [DEBUG] [app] Starting request: urn:request:69eb888e-ce01-4155-ab28-a3d95682c974 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:52,636 [244] [DEBUG] [app] Ending request: urn:request:ddda494e-c9e1-4de0-8b53-6322547d1d38 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ddda494e-c9e1-4de0-8b53-6322547d1d38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:52,637 [243] [DEBUG] [app] Ending request: urn:request:69eb888e-ce01-4155-ab28-a3d95682c974 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:69eb888e-ce01-4155-ab28-a3d95682c974', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:52,648 [241] [DEBUG] [app] Ending request: urn:request:3469f1fb-ed89-4b4b-ae51-0835972b66ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3469f1fb-ed89-4b4b-ae51-0835972b66ea', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:52,648 [239] [DEBUG] [app] Ending request: urn:request:fa6589ed-907b-4578-94cc-cd92be2b77c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fa6589ed-907b-4578-94cc-cd92be2b77c5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:13:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:13:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:09.230038+00:00 (in 16.392256 seconds) securityworker stdout | 2023-01-17 06:13:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:22 UTC)" (scheduled at 2023-01-17 06:13:52.837365+00:00) securityworker stdout | 2023-01-17 06:13:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:22 UTC)" executed successfully builder stdout | 2023-01-17 06:13:56,594 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:56,602 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:56,602 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:56,608 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:58.735066+00:00 (in 1.799546 seconds) notificationworker stdout | 2023-01-17 06:13:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:18:56 UTC)" (scheduled at 2023-01-17 06:13:56.935055+00:00) notificationworker stdout | 2023-01-17 06:13:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 13, 56, 935873), 'notification/%']) notificationworker stdout | 2023-01-17 06:13:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 13, 56, 935873), True, datetime.datetime(2023, 1, 17, 6, 13, 56, 935873), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:13:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 13, 56, 935873), True, datetime.datetime(2023, 1, 17, 6, 13, 56, 935873), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 13, 56, 935873), 'notification/%']) notificationworker stdout | 2023-01-17 06:13:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:18:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:13:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:13:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:27.226424+00:00 (in 29.999596 seconds) buildlogsarchiver stdout | 2023-01-17 06:13:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:27 UTC)" (scheduled at 2023-01-17 06:13:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:13:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 13, 57, 227170), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:13:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:13:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:13:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:13:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:08.735066+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 06:13:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:08 UTC)" (scheduled at 2023-01-17 06:13:58.735066+00:00) notificationworker stdout | 2023-01-17 06:13:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 58, 735874), True, datetime.datetime(2023, 1, 17, 6, 13, 58, 735874), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:13:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:13:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:15.931994+00:00 (in 16.594138 seconds) namespacegcworker stdout | 2023-01-17 06:13:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:59 UTC)" (scheduled at 2023-01-17 06:13:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:13:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:13:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 59, 338379), True, datetime.datetime(2023, 1, 17, 6, 13, 59, 338379), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:13:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:13:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:13:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:14:02,620 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:14:02,935 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:14:03,087 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:14:03,907 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:14:04,058 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:14:05,244 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:14:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:14:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:35.848675+00:00 (in 29.999468 seconds) gcworker stdout | 2023-01-17 06:14:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:35 UTC)" (scheduled at 2023-01-17 06:14:05.848675+00:00) gcworker stdout | 2023-01-17 06:14:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726445849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:14:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:14:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:14:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:35 UTC)" executed successfully builder stdout | 2023-01-17 06:14:05,973 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:14:06,016 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:14:06,386 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:14:06,533 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:14:06,535 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:14:06,537 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:14:06,618 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:06,627 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:06,627 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:06,633 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:14:07,372 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:14:07,426 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:14:07,625 [244] [DEBUG] [app] Starting request: urn:request:e86f4cec-7c61-409b-b18b-ad99a5d39a1e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:07,625 [241] [DEBUG] [app] Starting request: urn:request:551b2e6f-dd39-4d59-8e9c-e40258e47a38 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:07,630 [232] [DEBUG] [app] Starting request: urn:request:003481e5-1356-4566-8ebc-b4f04be177dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:07,630 [232] [DEBUG] [app] Ending request: urn:request:003481e5-1356-4566-8ebc-b4f04be177dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:003481e5-1356-4566-8ebc-b4f04be177dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:07,630 [233] [DEBUG] [app] Starting request: urn:request:e237b7a7-64b8-427c-9a78-d57b8abd4b62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:14:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:14:07,632 [233] [DEBUG] [app] Ending request: urn:request:e237b7a7-64b8-427c-9a78-d57b8abd4b62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e237b7a7-64b8-427c-9a78-d57b8abd4b62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 06:14:07,632 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:07,635 [241] [DEBUG] [app] Starting request: urn:request:67778abf-e0a3-4cc7-afc6-af119d88ae3c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:07,636 [241] [DEBUG] [app] Ending request: urn:request:67778abf-e0a3-4cc7-afc6-af119d88ae3c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:67778abf-e0a3-4cc7-afc6-af119d88ae3c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:14:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:07,637 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:07,638 [239] [DEBUG] [app] Starting request: urn:request:4a128cf1-61af-48fc-9869-cd93e7d15715 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:07,638 [239] [DEBUG] [app] Ending request: urn:request:4a128cf1-61af-48fc-9869-cd93e7d15715 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a128cf1-61af-48fc-9869-cd93e7d15715', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:07,638 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:07,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:14:07,639 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:07,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:07,647 [244] [DEBUG] [app] Ending request: urn:request:e86f4cec-7c61-409b-b18b-ad99a5d39a1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e86f4cec-7c61-409b-b18b-ad99a5d39a1e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:07,647 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:14:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:14:07,648 [241] [DEBUG] [app] Ending request: urn:request:551b2e6f-dd39-4d59-8e9c-e40258e47a38 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:551b2e6f-dd39-4d59-8e9c-e40258e47a38', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:07,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) chunkcleanupworker stdout | 2023-01-17 06:14:08,200 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:14:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:18.735066+00:00 (in 9.999600 seconds) notificationworker stdout | 2023-01-17 06:14:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:18 UTC)" (scheduled at 2023-01-17 06:14:08.735066+00:00) notificationworker stdout | 2023-01-17 06:14:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 8, 735806), True, datetime.datetime(2023, 1, 17, 6, 14, 8, 735806), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:08,748 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:08,748 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:08,748 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:14:09,032 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:14:09,034 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:14:09,036 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:14:09,039 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:14:09,041 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:14:09,043 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:14:09,048 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:14:09,052 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:14:09,054 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:14:09,062 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:14:09,065 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:14:09,067 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:14:09,069 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:14:09,072 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:14:09,106 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:14:09,109 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,128 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,128 [64] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 07:14:09.126527+00:00 (in 3599.998126 seconds) expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,128 [64] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 07:14:09 UTC)" (scheduled at 2023-01-17 06:14:09.126527+00:00) expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,128 [64] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00 expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,129 [64] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2023, 1, 16, 6, 14, 9, 128885)]) expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,135 [64] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 06:14:09,135 [64] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 07:14:09 UTC)" executed successfully securityworker stdout | 2023-01-17 06:14:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:22.837365+00:00 (in 13.606867 seconds) securityworker stdout | 2023-01-17 06:14:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:39 UTC)" (scheduled at 2023-01-17 06:14:09.230038+00:00) securityworker stdout | 2023-01-17 06:14:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:14:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:14:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:09,240 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:09,240 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:14:09,240 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:14:09,240 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:14:09,240 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:14:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:14:10,182 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:14:10,839 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:14:11,612 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:14:11,634 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:14:11,726 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:14:11,755 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} namespacegcworker stdout | 2023-01-17 06:14:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:14:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:47.932490+00:00 (in 32.000023 seconds) namespacegcworker stdout | 2023-01-17 06:14:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:19:15 UTC)" (scheduled at 2023-01-17 06:14:15.931994+00:00) namespacegcworker stdout | 2023-01-17 06:14:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 14, 15, 932809), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:14:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 14, 15, 932809), True, datetime.datetime(2023, 1, 17, 6, 14, 15, 932809), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:14:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 14, 15, 932809), True, datetime.datetime(2023, 1, 17, 6, 14, 15, 932809), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 14, 15, 932809), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:14:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:14:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:19:15 UTC)" executed successfully builder stdout | 2023-01-17 06:14:16,643 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:16,652 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:16,652 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:16,658 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:28.735066+00:00 (in 9.999592 seconds) notificationworker stdout | 2023-01-17 06:14:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:28 UTC)" (scheduled at 2023-01-17 06:14:18.735066+00:00) notificationworker stdout | 2023-01-17 06:14:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 18, 735890), True, datetime.datetime(2023, 1, 17, 6, 14, 18, 735890), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:14:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:14:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:25.334138+00:00 (in 5.091430 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:14:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:20 UTC)" (scheduled at 2023-01-17 06:14:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:14:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:14:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 20, 243109), True, datetime.datetime(2023, 1, 17, 6, 14, 20, 243109), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:14:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:14:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:14:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:14:22,625 [243] [DEBUG] [app] Starting request: urn:request:e54f0feb-0756-4a07-8291-d16a85e74004 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:22,625 [244] [DEBUG] [app] Starting request: urn:request:c0e053fb-1467-46df-9626-99aa5fea4331 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:22,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,628 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:22,630 [232] [DEBUG] [app] Starting request: urn:request:60deaeb6-03e6-40be-8173-575fcfad6a06 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:22,630 [240] [DEBUG] [app] Starting request: urn:request:3d2c76b8-650a-43a5-b902-53996e5b0e65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:22,630 [232] [DEBUG] [app] Ending request: urn:request:60deaeb6-03e6-40be-8173-575fcfad6a06 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60deaeb6-03e6-40be-8173-575fcfad6a06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:22,631 [240] [DEBUG] [app] Ending request: urn:request:3d2c76b8-650a-43a5-b902-53996e5b0e65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d2c76b8-650a-43a5-b902-53996e5b0e65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:14:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:14:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:14:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:22,635 [239] [DEBUG] [app] Starting request: urn:request:eea1e8bc-60ca-4c56-957b-c1191f6d5225 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:22,635 [241] [DEBUG] [app] Starting request: urn:request:61224cd6-f3cd-43cb-843a-da9a34e5c9d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:22,635 [241] [DEBUG] [app] Ending request: urn:request:61224cd6-f3cd-43cb-843a-da9a34e5c9d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61224cd6-f3cd-43cb-843a-da9a34e5c9d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:22,636 [239] [DEBUG] [app] Ending request: urn:request:eea1e8bc-60ca-4c56-957b-c1191f6d5225 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eea1e8bc-60ca-4c56-957b-c1191f6d5225', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:22,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:22,647 [244] [DEBUG] [app] Ending request: urn:request:c0e053fb-1467-46df-9626-99aa5fea4331 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0e053fb-1467-46df-9626-99aa5fea4331', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:22,647 [243] [DEBUG] [app] Ending request: urn:request:e54f0feb-0756-4a07-8291-d16a85e74004 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e54f0feb-0756-4a07-8291-d16a85e74004', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:22,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:14:22,647 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:14:22,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:14:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:39.230038+00:00 (in 16.392196 seconds) securityworker stdout | 2023-01-17 06:14:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:52 UTC)" (scheduled at 2023-01-17 06:14:22.837365+00:00) securityworker stdout | 2023-01-17 06:14:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:14:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:14:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:48.830274+00:00 (in 24.187986 seconds) exportactionlogsworker stdout | 2023-01-17 06:14:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:24 UTC)" (scheduled at 2023-01-17 06:14:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:14:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:14:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:14:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:14:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:20.242198+00:00 (in 54.907572 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:14:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:25 UTC)" (scheduled at 2023-01-17 06:14:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:14:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:14:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:25 UTC)" executed successfully builder stdout | 2023-01-17 06:14:26,668 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:26,677 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:26,677 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:26,683 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:14:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:14:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:57.226424+00:00 (in 29.999570 seconds) buildlogsarchiver stdout | 2023-01-17 06:14:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:57 UTC)" (scheduled at 2023-01-17 06:14:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:14:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 14, 27, 227236), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:14:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:14:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:14:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:38.735066+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 06:14:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:38 UTC)" (scheduled at 2023-01-17 06:14:28.735066+00:00) notificationworker stdout | 2023-01-17 06:14:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 28, 735958), True, datetime.datetime(2023, 1, 17, 6, 14, 28, 735958), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:14:32,656 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:14:32,952 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:14:33,093 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:14:33,933 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:14:34,079 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:14:35,264 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:14:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:14:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:05.848675+00:00 (in 29.999464 seconds) gcworker stdout | 2023-01-17 06:14:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:05 UTC)" (scheduled at 2023-01-17 06:14:35.848675+00:00) gcworker stdout | 2023-01-17 06:14:35,849 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 06:14:35,857 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726475856, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:14:35,860 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:14:35,860 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:14:35,860 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:05 UTC)" executed successfully builder stdout | 2023-01-17 06:14:36,009 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:14:36,034 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:14:36,422 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:14:36,540 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:14:36,542 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:14:36,545 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:14:36,693 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:36,702 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:36,702 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:36,707 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:14:37,408 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:14:37,450 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:14:37,625 [239] [DEBUG] [app] Starting request: urn:request:38822166-1873-4a87-b581-4621fe265de2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:37,625 [241] [DEBUG] [app] Starting request: urn:request:b76905b1-5ba9-42dc-8251-bce08dd08500 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:37,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:37,630 [240] [DEBUG] [app] Starting request: urn:request:6e8c366b-68d0-4d53-ac28-ec0b9b228115 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:37,631 [232] [DEBUG] [app] Starting request: urn:request:409f924b-8794-4542-a878-c3f1f7a4e7e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:37,631 [240] [DEBUG] [app] Ending request: urn:request:6e8c366b-68d0-4d53-ac28-ec0b9b228115 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e8c366b-68d0-4d53-ac28-ec0b9b228115', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:14:37,631 [232] [DEBUG] [app] Ending request: urn:request:409f924b-8794-4542-a878-c3f1f7a4e7e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:409f924b-8794-4542-a878-c3f1f7a4e7e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:14:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:14:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:37,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:37,636 [239] [DEBUG] [app] Starting request: urn:request:d3c5f548-1130-4653-8ea5-36081d1dc76c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:37,636 [244] [DEBUG] [app] Starting request: urn:request:b2f8ead6-8667-42dd-a04b-56680168b4aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:37,636 [239] [DEBUG] [app] Ending request: urn:request:d3c5f548-1130-4653-8ea5-36081d1dc76c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3c5f548-1130-4653-8ea5-36081d1dc76c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:37,636 [244] [DEBUG] [app] Ending request: urn:request:b2f8ead6-8667-42dd-a04b-56680168b4aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2f8ead6-8667-42dd-a04b-56680168b4aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:37,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:37,647 [239] [DEBUG] [app] Ending request: urn:request:38822166-1873-4a87-b581-4621fe265de2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38822166-1873-4a87-b581-4621fe265de2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:37,647 [241] [DEBUG] [app] Ending request: urn:request:b76905b1-5ba9-42dc-8251-bce08dd08500 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b76905b1-5ba9-42dc-8251-bce08dd08500', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:37,647 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:14:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:14:38,235 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:14:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:43.935629+00:00 (in 5.200122 seconds) notificationworker stdout | 2023-01-17 06:14:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:48 UTC)" (scheduled at 2023-01-17 06:14:38.735066+00:00) notificationworker stdout | 2023-01-17 06:14:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 38, 735858), True, datetime.datetime(2023, 1, 17, 6, 14, 38, 735858), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:14:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:14:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:48.738458+00:00 (in 9.707209 seconds) repositorygcworker stdout | 2023-01-17 06:14:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:39 UTC)" (scheduled at 2023-01-17 06:14:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:14:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:14:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 39, 31655), True, datetime.datetime(2023, 1, 17, 6, 14, 39, 31655), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:14:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:14:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:14:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:14:39,040 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:14:39,043 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:14:39,046 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:14:39,048 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:14:39,051 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:14:39,053 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:14:39,058 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:14:39,060 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:14:39,062 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:14:39,068 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:14:39,073 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:14:39,075 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:14:39,078 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:14:39,080 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:14:39,124 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:14:39,132 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} securityworker stdout | 2023-01-17 06:14:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:52.837365+00:00 (in 13.606854 seconds) securityworker stdout | 2023-01-17 06:14:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:09 UTC)" (scheduled at 2023-01-17 06:14:39.230038+00:00) securityworker stdout | 2023-01-17 06:14:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:14:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:14:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:14:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:14:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:14:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:14:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:14:40,213 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:14:40,874 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:14:41,638 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:14:41,666 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:14:41,738 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:14:41,775 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:14:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:48.735066+00:00 (in 4.798983 seconds) notificationworker stdout | 2023-01-17 06:14:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:43 UTC)" (scheduled at 2023-01-17 06:14:43.935629+00:00) notificationworker stdout | 2023-01-17 06:14:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:14:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:43 UTC)" executed successfully builder stdout | 2023-01-17 06:14:46,715 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:46,724 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:46,725 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:46,730 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:14:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:14:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:59.337413+00:00 (in 11.404475 seconds) namespacegcworker stdout | 2023-01-17 06:14:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:47 UTC)" (scheduled at 2023-01-17 06:14:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:14:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:14:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:58.735066+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 06:14:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:58 UTC)" (scheduled at 2023-01-17 06:14:48.735066+00:00) notificationworker stdout | 2023-01-17 06:14:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 48, 735867), True, datetime.datetime(2023, 1, 17, 6, 14, 48, 735867), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:14:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:14:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:39.030814+00:00 (in 50.291911 seconds) repositorygcworker stdout | 2023-01-17 06:14:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:48 UTC)" (scheduled at 2023-01-17 06:14:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:14:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:14:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:14:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:14:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:24.641816+00:00 (in 35.811071 seconds) exportactionlogsworker stdout | 2023-01-17 06:14:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:48 UTC)" (scheduled at 2023-01-17 06:14:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:14:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:14:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 48, 831165), True, datetime.datetime(2023, 1, 17, 6, 14, 48, 831165), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:14:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:14:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:14:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:14:52,626 [239] [DEBUG] [app] Starting request: urn:request:14b1f50b-1732-4061-b673-e7dce9ff318a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:52,626 [241] [DEBUG] [app] Starting request: urn:request:7cef05a1-35ee-4b78-a580-6ecd5c367dc2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:14:52,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:52,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:52,631 [232] [DEBUG] [app] Starting request: urn:request:67f5711b-a9a0-4c41-b45d-ea6271511c55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:52,631 [240] [DEBUG] [app] Starting request: urn:request:b422b2da-5166-461b-8e95-672f8af20ca3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:52,632 [232] [DEBUG] [app] Ending request: urn:request:67f5711b-a9a0-4c41-b45d-ea6271511c55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:67f5711b-a9a0-4c41-b45d-ea6271511c55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:52,632 [240] [DEBUG] [app] Ending request: urn:request:b422b2da-5166-461b-8e95-672f8af20ca3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b422b2da-5166-461b-8e95-672f8af20ca3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:52,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:14:52,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:52,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:14:52,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:52,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:52,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:52,637 [244] [DEBUG] [app] Starting request: urn:request:2048eec0-6688-447f-aa13-6126fd88ee08 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:52,637 [241] [DEBUG] [app] Starting request: urn:request:31651c69-518b-459f-9f9e-e095cfed9535 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:52,637 [244] [DEBUG] [app] Ending request: urn:request:2048eec0-6688-447f-aa13-6126fd88ee08 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2048eec0-6688-447f-aa13-6126fd88ee08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:52,637 [241] [DEBUG] [app] Ending request: urn:request:31651c69-518b-459f-9f9e-e095cfed9535 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31651c69-518b-459f-9f9e-e095cfed9535', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:14:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:52,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:52,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:52,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:52,643 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:52,649 [241] [DEBUG] [app] Ending request: urn:request:7cef05a1-35ee-4b78-a580-6ecd5c367dc2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7cef05a1-35ee-4b78-a580-6ecd5c367dc2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:52,649 [239] [DEBUG] [app] Ending request: urn:request:14b1f50b-1732-4061-b673-e7dce9ff318a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14b1f50b-1732-4061-b673-e7dce9ff318a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:52,649 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:52,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:52,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) gunicorn-web stdout | 2023-01-17 06:14:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:14:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:09.230038+00:00 (in 16.392227 seconds) securityworker stdout | 2023-01-17 06:14:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:22 UTC)" (scheduled at 2023-01-17 06:14:52.837365+00:00) securityworker stdout | 2023-01-17 06:14:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:52,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:52,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:22 UTC)" executed successfully builder stdout | 2023-01-17 06:14:56,738 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:56,746 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:56,747 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:56,752 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:14:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:14:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:27.226424+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 06:14:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:27 UTC)" (scheduled at 2023-01-17 06:14:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:14:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 14, 57, 227266), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:14:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:14:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:14:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:08.735066+00:00 (in 9.999575 seconds) notificationworker stdout | 2023-01-17 06:14:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:08 UTC)" (scheduled at 2023-01-17 06:14:58.735066+00:00) notificationworker stdout | 2023-01-17 06:14:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 58, 735941), True, datetime.datetime(2023, 1, 17, 6, 14, 58, 735941), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:58,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:58,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:14:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:14:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:47.932490+00:00 (in 48.594664 seconds) namespacegcworker stdout | 2023-01-17 06:14:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:59 UTC)" (scheduled at 2023-01-17 06:14:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:14:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:14:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 59, 338514), True, datetime.datetime(2023, 1, 17, 6, 14, 59, 338514), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:14:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:14:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:14:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:15:02,669 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:15:02,987 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:15:03,124 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:15:03,959 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:15:04,114 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:15:05,281 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:15:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:15:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:35.848675+00:00 (in 29.999533 seconds) gcworker stdout | 2023-01-17 06:15:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:35 UTC)" (scheduled at 2023-01-17 06:15:05.848675+00:00) gcworker stdout | 2023-01-17 06:15:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726505849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:15:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:15:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:35 UTC)" executed successfully builder stdout | 2023-01-17 06:15:06,044 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:15:06,065 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:15:06,455 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:15:06,547 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:15:06,550 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:15:06,552 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:15:06,762 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:06,771 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:06,771 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:06,777 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:15:07,434 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:15:07,468 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:15:07,625 [244] [DEBUG] [app] Starting request: urn:request:996fd3fe-7b4c-4e08-9fc3-0fa0b8d86ec4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:07,625 [243] [DEBUG] [app] Starting request: urn:request:69a54a0d-0fad-4bf5-9a84-dad51d259cc2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:07,630 [240] [DEBUG] [app] Starting request: urn:request:6c1c5205-8176-430b-8359-87153d8fd3e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:07,630 [235] [DEBUG] [app] Starting request: urn:request:a541ae72-4c67-4cb6-ae7d-82949c59a519 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:07,630 [240] [DEBUG] [app] Ending request: urn:request:6c1c5205-8176-430b-8359-87153d8fd3e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c1c5205-8176-430b-8359-87153d8fd3e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:15:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:15:07,631 [235] [DEBUG] [app] Ending request: urn:request:a541ae72-4c67-4cb6-ae7d-82949c59a519 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a541ae72-4c67-4cb6-ae7d-82949c59a519', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:15:07,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:15:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:07,635 [241] [DEBUG] [app] Starting request: urn:request:9f724dea-d37a-4357-b458-335b66a94a8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:07,635 [239] [DEBUG] [app] Starting request: urn:request:68b87211-f2f0-471a-9c89-49c83a0b585f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:07,636 [241] [DEBUG] [app] Ending request: urn:request:9f724dea-d37a-4357-b458-335b66a94a8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f724dea-d37a-4357-b458-335b66a94a8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:07,636 [239] [DEBUG] [app] Ending request: urn:request:68b87211-f2f0-471a-9c89-49c83a0b585f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68b87211-f2f0-471a-9c89-49c83a0b585f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:07,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:15:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:07,647 [244] [DEBUG] [app] Ending request: urn:request:996fd3fe-7b4c-4e08-9fc3-0fa0b8d86ec4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:996fd3fe-7b4c-4e08-9fc3-0fa0b8d86ec4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:07,647 [243] [DEBUG] [app] Ending request: urn:request:69a54a0d-0fad-4bf5-9a84-dad51d259cc2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69a54a0d-0fad-4bf5-9a84-dad51d259cc2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:07,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:07,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:15:07,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:15:08,271 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:15:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:18.735066+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 06:15:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:18 UTC)" (scheduled at 2023-01-17 06:15:08.735066+00:00) notificationworker stdout | 2023-01-17 06:15:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 8, 735868), True, datetime.datetime(2023, 1, 17, 6, 15, 8, 735868), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:15:09,047 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:15:09,053 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:15:09,056 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:15:09,059 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:15:09,061 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:15:09,064 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:15:09,066 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:15:09,069 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:15:09,071 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:15:09,074 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:15:09,079 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:15:09,083 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:15:09,086 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:15:09,088 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:15:09,159 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:15:09,166 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} securityworker stdout | 2023-01-17 06:15:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:22.837365+00:00 (in 13.606895 seconds) securityworker stdout | 2023-01-17 06:15:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:39 UTC)" (scheduled at 2023-01-17 06:15:09.230038+00:00) securityworker stdout | 2023-01-17 06:15:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:15:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:15:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:15:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:15:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:15:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:15:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:15:10,249 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:15:10,892 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:15:11,676 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:15:11,702 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:15:11,774 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:15:11,810 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:15:16,787 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:16,796 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:16,796 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:16,802 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:28.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 06:15:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:28 UTC)" (scheduled at 2023-01-17 06:15:18.735066+00:00) notificationworker stdout | 2023-01-17 06:15:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 18, 735917), True, datetime.datetime(2023, 1, 17, 6, 15, 18, 735917), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:15:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:15:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:25.334138+00:00 (in 5.091466 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:15:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:20 UTC)" (scheduled at 2023-01-17 06:15:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:15:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:15:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 20, 243122), True, datetime.datetime(2023, 1, 17, 6, 15, 20, 243122), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:15:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:15:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:15:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:15:22,625 [239] [DEBUG] [app] Starting request: urn:request:97675ea9-d68a-486d-bfe8-013bed055c75 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:22,626 [243] [DEBUG] [app] Starting request: urn:request:caf148b7-f211-4a14-b65a-89112eff6106 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:22,630 [240] [DEBUG] [app] Starting request: urn:request:f789dbeb-b556-486f-b753-232e9b933268 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,631 [237] [DEBUG] [app] Starting request: urn:request:65409eab-ba5e-4f6f-8f08-23fe864a6c45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,631 [240] [DEBUG] [app] Ending request: urn:request:f789dbeb-b556-486f-b753-232e9b933268 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f789dbeb-b556-486f-b753-232e9b933268', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,631 [237] [DEBUG] [app] Ending request: urn:request:65409eab-ba5e-4f6f-8f08-23fe864a6c45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:65409eab-ba5e-4f6f-8f08-23fe864a6c45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:15:22,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:22,636 [244] [DEBUG] [app] Starting request: urn:request:00328e0e-1c0f-4eb6-b4a9-3e635f76506b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:22,637 [243] [DEBUG] [app] Starting request: urn:request:8ab7989d-1d09-4702-a4a8-e4bfa2dac829 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:22,637 [244] [DEBUG] [app] Ending request: urn:request:00328e0e-1c0f-4eb6-b4a9-3e635f76506b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00328e0e-1c0f-4eb6-b4a9-3e635f76506b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:22,637 [243] [DEBUG] [app] Ending request: urn:request:8ab7989d-1d09-4702-a4a8-e4bfa2dac829 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ab7989d-1d09-4702-a4a8-e4bfa2dac829', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:22,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:22,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:22,648 [239] [DEBUG] [app] Ending request: urn:request:97675ea9-d68a-486d-bfe8-013bed055c75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97675ea9-d68a-486d-bfe8-013bed055c75', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:22,648 [243] [DEBUG] [app] Ending request: urn:request:caf148b7-f211-4a14-b65a-89112eff6106 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:caf148b7-f211-4a14-b65a-89112eff6106', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:22,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:15:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:15:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:15:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:39.230038+00:00 (in 16.392229 seconds) securityworker stdout | 2023-01-17 06:15:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:52 UTC)" (scheduled at 2023-01-17 06:15:22.837365+00:00) securityworker stdout | 2023-01-17 06:15:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:22,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:15:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:15:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:48.830274+00:00 (in 24.187973 seconds) exportactionlogsworker stdout | 2023-01-17 06:15:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:24 UTC)" (scheduled at 2023-01-17 06:15:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:15:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:15:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:15:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:15:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:20.242198+00:00 (in 54.907559 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:15:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:25 UTC)" (scheduled at 2023-01-17 06:15:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:15:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:15:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:25 UTC)" executed successfully builder stdout | 2023-01-17 06:15:26,811 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:26,819 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:26,819 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:26,825 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:15:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:15:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:57.226424+00:00 (in 29.999542 seconds) buildlogsarchiver stdout | 2023-01-17 06:15:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:57 UTC)" (scheduled at 2023-01-17 06:15:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:15:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 15, 27, 227257), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:15:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:15:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:15:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:38.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 06:15:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:38 UTC)" (scheduled at 2023-01-17 06:15:28.735066+00:00) notificationworker stdout | 2023-01-17 06:15:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 28, 735895), True, datetime.datetime(2023, 1, 17, 6, 15, 28, 735895), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:15:32,681 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:15:32,994 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:15:33,141 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:15:33,994 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:15:34,148 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:15:35,289 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:15:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:15:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:05.848675+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 06:15:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:05 UTC)" (scheduled at 2023-01-17 06:15:35.848675+00:00) gcworker stdout | 2023-01-17 06:15:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726535849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:15:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:15:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:05 UTC)" executed successfully builder stdout | 2023-01-17 06:15:36,051 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:15:36,100 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:15:36,485 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:15:36,554 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:15:36,556 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:15:36,558 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:15:36,831 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:36,840 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:36,840 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:36,846 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:15:37,470 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:15:37,474 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:15:37,625 [239] [DEBUG] [app] Starting request: urn:request:a5b52cf4-49bb-49e3-9f55-074ec75f93e9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:37,625 [244] [DEBUG] [app] Starting request: urn:request:f0500beb-6fa3-4443-89bd-50bc123d871c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:37,629 [232] [DEBUG] [app] Starting request: urn:request:82a916be-f024-43db-8758-4b8a4ecca534 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:37,630 [232] [DEBUG] [app] Ending request: urn:request:82a916be-f024-43db-8758-4b8a4ecca534 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82a916be-f024-43db-8758-4b8a4ecca534', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:37,630 [235] [DEBUG] [app] Starting request: urn:request:0edae336-d6b6-4ff0-8f2a-ec17d8a421fb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:37,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:15:37,630 [235] [DEBUG] [app] Ending request: urn:request:0edae336-d6b6-4ff0-8f2a-ec17d8a421fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0edae336-d6b6-4ff0-8f2a-ec17d8a421fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:37,630 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:15:37,631 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:37,634 [243] [DEBUG] [app] Starting request: urn:request:ccdb5602-8ce0-4de3-be4a-67d5428e91b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:37,635 [239] [DEBUG] [app] Starting request: urn:request:ccde8de6-7bb4-4f68-ab46-b16601823a34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:37,635 [243] [DEBUG] [app] Ending request: urn:request:ccdb5602-8ce0-4de3-be4a-67d5428e91b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccdb5602-8ce0-4de3-be4a-67d5428e91b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:37,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:37,635 [239] [DEBUG] [app] Ending request: urn:request:ccde8de6-7bb4-4f68-ab46-b16601823a34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccde8de6-7bb4-4f68-ab46-b16601823a34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:15:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:37,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:37,646 [244] [DEBUG] [app] Ending request: urn:request:f0500beb-6fa3-4443-89bd-50bc123d871c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0500beb-6fa3-4443-89bd-50bc123d871c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:37,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 06:15:37,649 [239] [DEBUG] [app] Ending request: urn:request:a5b52cf4-49bb-49e3-9f55-074ec75f93e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5b52cf4-49bb-49e3-9f55-074ec75f93e9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:37,649 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:37,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) chunkcleanupworker stdout | 2023-01-17 06:15:38,293 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:15:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:43.935629+00:00 (in 5.200159 seconds) notificationworker stdout | 2023-01-17 06:15:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:48 UTC)" (scheduled at 2023-01-17 06:15:38.735066+00:00) notificationworker stdout | 2023-01-17 06:15:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 38, 735864), True, datetime.datetime(2023, 1, 17, 6, 15, 38, 735864), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:15:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:15:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:48.738458+00:00 (in 9.707247 seconds) repositorygcworker stdout | 2023-01-17 06:15:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:39 UTC)" (scheduled at 2023-01-17 06:15:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:15:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:15:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 39, 31629), True, datetime.datetime(2023, 1, 17, 6, 15, 39, 31629), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:15:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:15:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:15:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:15:39,055 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:15:39,062 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:15:39,065 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:15:39,068 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:15:39,071 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:15:39,073 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:15:39,077 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:15:39,080 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:15:39,082 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:15:39,085 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:15:39,088 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:15:39,091 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:15:39,094 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-web stdout | 2023-01-17 06:15:39,096 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:15:39,165 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:15:39,177 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} securityworker stdout | 2023-01-17 06:15:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:52.837365+00:00 (in 13.606864 seconds) securityworker stdout | 2023-01-17 06:15:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:09 UTC)" (scheduled at 2023-01-17 06:15:39.230038+00:00) securityworker stdout | 2023-01-17 06:15:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:15:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:15:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:15:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:09 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:15:40,267 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:15:40,928 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:15:41,709 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:15:41,730 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:15:41,782 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:15:41,819 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:15:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:48.735066+00:00 (in 4.798976 seconds) notificationworker stdout | 2023-01-17 06:15:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:43 UTC)" (scheduled at 2023-01-17 06:15:43.935629+00:00) notificationworker stdout | 2023-01-17 06:15:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:15:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:43 UTC)" executed successfully builder stdout | 2023-01-17 06:15:46,856 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:46,865 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:46,865 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:46,871 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:15:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:15:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:59.337413+00:00 (in 11.404498 seconds) namespacegcworker stdout | 2023-01-17 06:15:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:47 UTC)" (scheduled at 2023-01-17 06:15:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:15:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:15:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:58.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 06:15:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:58 UTC)" (scheduled at 2023-01-17 06:15:48.735066+00:00) notificationworker stdout | 2023-01-17 06:15:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 48, 736032), True, datetime.datetime(2023, 1, 17, 6, 15, 48, 736032), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:15:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:15:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:39.030814+00:00 (in 50.291877 seconds) repositorygcworker stdout | 2023-01-17 06:15:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:48 UTC)" (scheduled at 2023-01-17 06:15:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:15:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:15:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:15:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:15:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:24.641816+00:00 (in 35.811115 seconds) exportactionlogsworker stdout | 2023-01-17 06:15:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:48 UTC)" (scheduled at 2023-01-17 06:15:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:15:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:15:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 48, 831071), True, datetime.datetime(2023, 1, 17, 6, 15, 48, 831071), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:15:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:15:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:15:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:15:52,625 [239] [DEBUG] [app] Starting request: urn:request:4d7f5230-c08e-408f-a2fd-e91eae75d6ec (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:52,625 [244] [DEBUG] [app] Starting request: urn:request:db55d81d-577d-4fc6-b885-bee5f16797b4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:15:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:52,630 [240] [DEBUG] [app] Starting request: urn:request:2a1d0033-507f-4eb7-af1c-238ef7a9804e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:52,631 [240] [DEBUG] [app] Ending request: urn:request:2a1d0033-507f-4eb7-af1c-238ef7a9804e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a1d0033-507f-4eb7-af1c-238ef7a9804e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:15:52,631 [235] [DEBUG] [app] Starting request: urn:request:0a0da716-a598-4f27-8247-bdd4d14f37fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:15:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:15:52,631 [235] [DEBUG] [app] Ending request: urn:request:0a0da716-a598-4f27-8247-bdd4d14f37fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a0da716-a598-4f27-8247-bdd4d14f37fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:15:52,632 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:52,635 [244] [DEBUG] [app] Starting request: urn:request:a951bcc4-8166-49f3-b231-cce317d2d048 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:52,635 [244] [DEBUG] [app] Ending request: urn:request:a951bcc4-8166-49f3-b231-cce317d2d048 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a951bcc4-8166-49f3-b231-cce317d2d048', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:52,637 [243] [DEBUG] [app] Starting request: urn:request:a59008f2-4683-450c-a934-7dc314ce9b11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:52,638 [243] [DEBUG] [app] Ending request: urn:request:a59008f2-4683-450c-a934-7dc314ce9b11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a59008f2-4683-450c-a934-7dc314ce9b11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:52,647 [239] [DEBUG] [app] Ending request: urn:request:4d7f5230-c08e-408f-a2fd-e91eae75d6ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d7f5230-c08e-408f-a2fd-e91eae75d6ec', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:52,647 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:15:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:15:52,648 [244] [DEBUG] [app] Ending request: urn:request:db55d81d-577d-4fc6-b885-bee5f16797b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db55d81d-577d-4fc6-b885-bee5f16797b4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:52,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:15:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:15:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:09.230038+00:00 (in 16.392262 seconds) securityworker stdout | 2023-01-17 06:15:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:22 UTC)" (scheduled at 2023-01-17 06:15:52.837365+00:00) securityworker stdout | 2023-01-17 06:15:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:22 UTC)" executed successfully builder stdout | 2023-01-17 06:15:56,880 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:56,889 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:56,889 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:56,896 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:15:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:15:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:27.226424+00:00 (in 29.999551 seconds) buildlogsarchiver stdout | 2023-01-17 06:15:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:27 UTC)" (scheduled at 2023-01-17 06:15:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:15:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 15, 57, 227186), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:15:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:15:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:15:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:08.735066+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 06:15:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:08 UTC)" (scheduled at 2023-01-17 06:15:58.735066+00:00) notificationworker stdout | 2023-01-17 06:15:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 58, 735944), True, datetime.datetime(2023, 1, 17, 6, 15, 58, 735944), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:15:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:15:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:47.932490+00:00 (in 48.594649 seconds) namespacegcworker stdout | 2023-01-17 06:15:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:59 UTC)" (scheduled at 2023-01-17 06:15:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:15:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:15:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 59, 338264), True, datetime.datetime(2023, 1, 17, 6, 15, 59, 338264), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:15:59,355 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:15:59,355 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:15:59,355 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:16:02,716 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:16:03,022 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:16:03,176 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:16:04,027 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:16:04,183 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:16:05,314 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:16:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:16:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:35.848675+00:00 (in 29.999503 seconds) gcworker stdout | 2023-01-17 06:16:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:35 UTC)" (scheduled at 2023-01-17 06:16:05.848675+00:00) gcworker stdout | 2023-01-17 06:16:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726565849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:16:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:16:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:35 UTC)" executed successfully builder stdout | 2023-01-17 06:16:06,086 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:16:06,135 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:16:06,514 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:16:06,561 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:16:06,564 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:16:06,566 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:16:06,906 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:06,915 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:06,915 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:06,920 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:16:07,481 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:16:07,505 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-web stdout | 2023-01-17 06:16:07,626 [244] [DEBUG] [app] Starting request: urn:request:7e8b0095-b489-4ec7-aaa4-19480e751542 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,626 [241] [DEBUG] [app] Starting request: urn:request:1fb46960-f700-46db-9d4a-7a8871110ed1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:07,631 [237] [DEBUG] [app] Starting request: urn:request:7b1f033b-0299-40eb-999e-5d93ff725622 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:07,631 [237] [DEBUG] [app] Ending request: urn:request:7b1f033b-0299-40eb-999e-5d93ff725622 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b1f033b-0299-40eb-999e-5d93ff725622', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:07,631 [242] [DEBUG] [app] Starting request: urn:request:4cae1144-823b-4665-b492-667459c0065a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:16:07,632 [242] [DEBUG] [app] Ending request: urn:request:4cae1144-823b-4665-b492-667459c0065a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4cae1144-823b-4665-b492-667459c0065a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:16:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:16:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:07,636 [239] [DEBUG] [app] Starting request: urn:request:465f67d5-b4cd-4945-8c0a-e871b4d07203 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:07,637 [239] [DEBUG] [app] Ending request: urn:request:465f67d5-b4cd-4945-8c0a-e871b4d07203 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:465f67d5-b4cd-4945-8c0a-e871b4d07203', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:07,637 [243] [DEBUG] [app] Starting request: urn:request:cf68a9cf-8388-498d-88ad-0e7f90443097 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:07,638 [243] [DEBUG] [app] Ending request: urn:request:cf68a9cf-8388-498d-88ad-0e7f90443097 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cf68a9cf-8388-498d-88ad-0e7f90443097', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:07,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:07,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:07,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:07,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:07,648 [241] [DEBUG] [app] Ending request: urn:request:1fb46960-f700-46db-9d4a-7a8871110ed1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1fb46960-f700-46db-9d4a-7a8871110ed1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:07,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:16:07,650 [244] [DEBUG] [app] Ending request: urn:request:7e8b0095-b489-4ec7-aaa4-19480e751542 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e8b0095-b489-4ec7-aaa4-19480e751542', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:07,651 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:07,651 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.027) chunkcleanupworker stdout | 2023-01-17 06:16:08,325 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:16:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:18.735066+00:00 (in 9.999406 seconds) notificationworker stdout | 2023-01-17 06:16:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:18 UTC)" (scheduled at 2023-01-17 06:16:08.735066+00:00) notificationworker stdout | 2023-01-17 06:16:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:08,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 8, 736150), True, datetime.datetime(2023, 1, 17, 6, 16, 8, 736150), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:16:09,066 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:16:09,071 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:16:09,074 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:16:09,076 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:16:09,079 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:16:09,081 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:16:09,085 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:16:09,088 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:16:09,090 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:16:09,093 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:16:09,095 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:16:09,097 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:16:09,101 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:16:09,104 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} logrotateworker stdout | 2023-01-17 06:16:09,186 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:16:09,188 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} securityworker stdout | 2023-01-17 06:16:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:22.837365+00:00 (in 13.606910 seconds) securityworker stdout | 2023-01-17 06:16:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:39 UTC)" (scheduled at 2023-01-17 06:16:09.230038+00:00) securityworker stdout | 2023-01-17 06:16:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:16:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:16:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:09,235 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:09,236 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:09,243 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:09,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:16:09,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:16:10,294 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:16:10,936 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:16:11,733 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:16:11,765 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:16:11,817 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:16:11,851 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:16:16,931 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:16,940 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:16,940 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:16,946 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:28.735066+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 06:16:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:28 UTC)" (scheduled at 2023-01-17 06:16:18.735066+00:00) notificationworker stdout | 2023-01-17 06:16:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 18, 735972), True, datetime.datetime(2023, 1, 17, 6, 16, 18, 735972), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:16:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:16:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:25.334138+00:00 (in 5.091504 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:16:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:20 UTC)" (scheduled at 2023-01-17 06:16:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:16:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:16:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 20, 243065), True, datetime.datetime(2023, 1, 17, 6, 16, 20, 243065), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:16:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:16:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:16:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:16:22,625 [244] [DEBUG] [app] Starting request: urn:request:07256811-76fc-47ea-a3c3-dbf3231d96aa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:22,625 [243] [DEBUG] [app] Starting request: urn:request:c1177f75-1301-4904-bbcc-35e7fad8ba11 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:22,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:22,630 [232] [DEBUG] [app] Starting request: urn:request:a6c6c8af-2291-4bda-96ef-6d942b38456e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:22,630 [237] [DEBUG] [app] Starting request: urn:request:6fa2c533-8a4e-4aa3-9216-ca3409fe5e8f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:22,630 [232] [DEBUG] [app] Ending request: urn:request:a6c6c8af-2291-4bda-96ef-6d942b38456e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a6c6c8af-2291-4bda-96ef-6d942b38456e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:22,630 [237] [DEBUG] [app] Ending request: urn:request:6fa2c533-8a4e-4aa3-9216-ca3409fe5e8f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6fa2c533-8a4e-4aa3-9216-ca3409fe5e8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:16:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:16:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:22,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:16:22,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:22,635 [239] [DEBUG] [app] Starting request: urn:request:4ed6ef08-c0a7-4d00-9080-50ab347d837a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:22,636 [241] [DEBUG] [app] Starting request: urn:request:89d104d9-9420-4b96-9676-b6d0a09ebac2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:22,636 [239] [DEBUG] [app] Ending request: urn:request:4ed6ef08-c0a7-4d00-9080-50ab347d837a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4ed6ef08-c0a7-4d00-9080-50ab347d837a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:22,636 [241] [DEBUG] [app] Ending request: urn:request:89d104d9-9420-4b96-9676-b6d0a09ebac2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89d104d9-9420-4b96-9676-b6d0a09ebac2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:22,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:22,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:16:22,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:22,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:22,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:22,647 [244] [DEBUG] [app] Ending request: urn:request:07256811-76fc-47ea-a3c3-dbf3231d96aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07256811-76fc-47ea-a3c3-dbf3231d96aa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:22,647 [243] [DEBUG] [app] Ending request: urn:request:c1177f75-1301-4904-bbcc-35e7fad8ba11 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1177f75-1301-4904-bbcc-35e7fad8ba11', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:22,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:22,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:16:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 06:16:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:39.230038+00:00 (in 16.392251 seconds) securityworker stdout | 2023-01-17 06:16:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:52 UTC)" (scheduled at 2023-01-17 06:16:22.837365+00:00) securityworker stdout | 2023-01-17 06:16:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:16:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:16:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:44.641497+00:00 (in 19.999280 seconds) exportactionlogsworker stdout | 2023-01-17 06:16:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:24 UTC)" (scheduled at 2023-01-17 06:16:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:16:24,643 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:16:24,643 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:16:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:16:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:39.333860+00:00 (in 13.999242 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:16:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:25 UTC)" (scheduled at 2023-01-17 06:16:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:16:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:16:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:25 UTC)" executed successfully builder stdout | 2023-01-17 06:16:26,951 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:26,960 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:26,960 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:26,965 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:16:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:16:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:57.226424+00:00 (in 29.999584 seconds) buildlogsarchiver stdout | 2023-01-17 06:16:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:57 UTC)" (scheduled at 2023-01-17 06:16:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:16:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 16, 27, 227198), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:16:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:16:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:16:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:38.735066+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 06:16:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:38 UTC)" (scheduled at 2023-01-17 06:16:28.735066+00:00) notificationworker stdout | 2023-01-17 06:16:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 28, 735970), True, datetime.datetime(2023, 1, 17, 6, 16, 28, 735970), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:16:32,743 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:16:33,041 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:16:33,212 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:16:34,043 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:16:34,218 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:16:35,331 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:16:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:16:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:05.848675+00:00 (in 29.999556 seconds) gcworker stdout | 2023-01-17 06:16:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:05 UTC)" (scheduled at 2023-01-17 06:16:35.848675+00:00) gcworker stdout | 2023-01-17 06:16:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726595849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:16:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:16:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:05 UTC)" executed successfully builder stdout | 2023-01-17 06:16:36,122 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:16:36,171 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:16:36,550 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:16:36,568 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:16:36,571 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:16:36,573 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:16:36,976 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:36,984 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:36,984 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:36,990 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:16:37,504 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:16:37,531 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-web stdout | 2023-01-17 06:16:37,624 [239] [DEBUG] [app] Starting request: urn:request:a689d3c2-68d9-48d2-9218-2243ee9a723a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:37,625 [244] [DEBUG] [app] Starting request: urn:request:6d74d410-dd78-4c78-8cd8-9f6f0c2ad380 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:37,630 [240] [DEBUG] [app] Starting request: urn:request:2b2d5cae-c7df-4174-9323-0e26534581a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:37,630 [232] [DEBUG] [app] Starting request: urn:request:fbe16962-b856-4729-8f75-9034cffb3671 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:37,630 [240] [DEBUG] [app] Ending request: urn:request:2b2d5cae-c7df-4174-9323-0e26534581a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2b2d5cae-c7df-4174-9323-0e26534581a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:37,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:16:37,630 [232] [DEBUG] [app] Ending request: urn:request:fbe16962-b856-4729-8f75-9034cffb3671 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fbe16962-b856-4729-8f75-9034cffb3671', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:37,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:16:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:16:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:37,635 [243] [DEBUG] [app] Starting request: urn:request:1cdb305f-4e9a-44ed-a242-08d56239e504 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:37,635 [241] [DEBUG] [app] Starting request: urn:request:a9fed48a-f61e-4cb2-a237-c35c33f220ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:37,635 [243] [DEBUG] [app] Ending request: urn:request:1cdb305f-4e9a-44ed-a242-08d56239e504 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1cdb305f-4e9a-44ed-a242-08d56239e504', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:37,636 [241] [DEBUG] [app] Ending request: urn:request:a9fed48a-f61e-4cb2-a237-c35c33f220ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9fed48a-f61e-4cb2-a237-c35c33f220ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:16:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:37,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:37,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:37,646 [244] [DEBUG] [app] Ending request: urn:request:6d74d410-dd78-4c78-8cd8-9f6f0c2ad380 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d74d410-dd78-4c78-8cd8-9f6f0c2ad380', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:37,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:37,646 [239] [DEBUG] [app] Ending request: urn:request:a689d3c2-68d9-48d2-9218-2243ee9a723a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a689d3c2-68d9-48d2-9218-2243ee9a723a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:37,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:16:37,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) chunkcleanupworker stdout | 2023-01-17 06:16:38,342 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:16:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:43.935629+00:00 (in 5.200094 seconds) notificationworker stdout | 2023-01-17 06:16:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:48 UTC)" (scheduled at 2023-01-17 06:16:38.735066+00:00) notificationworker stdout | 2023-01-17 06:16:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 38, 736019), True, datetime.datetime(2023, 1, 17, 6, 16, 38, 736019), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:16:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:16:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:48.738458+00:00 (in 9.707228 seconds) repositorygcworker stdout | 2023-01-17 06:16:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:39 UTC)" (scheduled at 2023-01-17 06:16:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:16:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:16:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 39, 31686), True, datetime.datetime(2023, 1, 17, 6, 16, 39, 31686), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:16:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:16:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:16:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:16:39,076 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:16:39,080 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:16:39,082 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:16:39,085 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:16:39,088 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:16:39,090 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:16:39,093 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:16:39,096 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:16:39,098 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:16:39,100 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:16:39,103 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:16:39,106 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:16:39,108 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:16:39,110 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} logrotateworker stdout | 2023-01-17 06:16:39,214 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:16:39,222 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} securityworker stdout | 2023-01-17 06:16:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:52.837365+00:00 (in 13.606827 seconds) securityworker stdout | 2023-01-17 06:16:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:09 UTC)" (scheduled at 2023-01-17 06:16:39.230038+00:00) securityworker stdout | 2023-01-17 06:16:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:16:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:16:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:16:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:16:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:16:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:20.242198+00:00 (in 40.907919 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:16:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:39 UTC)" (scheduled at 2023-01-17 06:16:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:16:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 16, 39, 334648), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:16:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 39, 334648), True, datetime.datetime(2023, 1, 17, 6, 16, 39, 334648), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:16:39,343 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 39, 334648), True, datetime.datetime(2023, 1, 17, 6, 16, 39, 334648), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 16, 39, 334648), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:16:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:16:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:39 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:16:40,310 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:16:40,971 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:16:41,770 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:16:41,800 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:16:41,852 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:16:41,886 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:16:43,936 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:48.735066+00:00 (in 4.798927 seconds) notificationworker stdout | 2023-01-17 06:16:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:43 UTC)" (scheduled at 2023-01-17 06:16:43.935629+00:00) notificationworker stdout | 2023-01-17 06:16:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:16:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:16:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:16:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:48.830274+00:00 (in 4.188371 seconds) exportactionlogsworker stdout | 2023-01-17 06:16:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:44 UTC)" (scheduled at 2023-01-17 06:16:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 06:16:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 16, 44, 642226), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:16:44,649 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 44, 642226), True, datetime.datetime(2023, 1, 17, 6, 16, 44, 642226), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:16:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 44, 642226), True, datetime.datetime(2023, 1, 17, 6, 16, 44, 642226), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 16, 44, 642226), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:16:44,653 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:16:44,653 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:44 UTC)" executed successfully builder stdout | 2023-01-17 06:16:47,000 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:47,008 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:47,008 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:47,014 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:16:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:16:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:59.337413+00:00 (in 11.404410 seconds) namespacegcworker stdout | 2023-01-17 06:16:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:47 UTC)" (scheduled at 2023-01-17 06:16:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:16:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:16:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:58.735066+00:00 (in 9.999594 seconds) notificationworker stdout | 2023-01-17 06:16:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:58 UTC)" (scheduled at 2023-01-17 06:16:48.735066+00:00) notificationworker stdout | 2023-01-17 06:16:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 48, 735830), True, datetime.datetime(2023, 1, 17, 6, 16, 48, 735830), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:16:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:16:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:04.738062+00:00 (in 15.999155 seconds) repositorygcworker stdout | 2023-01-17 06:16:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:48 UTC)" (scheduled at 2023-01-17 06:16:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:16:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:16:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:16:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:16:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:24.641816+00:00 (in 35.811103 seconds) exportactionlogsworker stdout | 2023-01-17 06:16:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:48 UTC)" (scheduled at 2023-01-17 06:16:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:16:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:16:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 48, 831098), True, datetime.datetime(2023, 1, 17, 6, 16, 48, 831098), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:16:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:16:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:16:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:16:52,625 [244] [DEBUG] [app] Starting request: urn:request:1c4d9df3-e832-4d1c-aa64-9e2af73fdd8c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:52,625 [239] [DEBUG] [app] Starting request: urn:request:7083b102-d89c-408c-994b-9d8baeb7a4b0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:16:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:52,630 [240] [DEBUG] [app] Starting request: urn:request:e5c40007-665e-4a0d-9842-e09145e374f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,630 [237] [DEBUG] [app] Starting request: urn:request:30e06803-b36b-49bf-af81-99543f257817 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,630 [240] [DEBUG] [app] Ending request: urn:request:e5c40007-665e-4a0d-9842-e09145e374f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5c40007-665e-4a0d-9842-e09145e374f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:16:52,631 [237] [DEBUG] [app] Ending request: urn:request:30e06803-b36b-49bf-af81-99543f257817 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30e06803-b36b-49bf-af81-99543f257817', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:16:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:52,636 [241] [DEBUG] [app] Starting request: urn:request:6d29539c-81c7-437f-867b-3bd9a5a4d95f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:52,636 [243] [DEBUG] [app] Starting request: urn:request:8cdea7af-caf7-4c11-a3c3-9a3692a3f860 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:52,636 [241] [DEBUG] [app] Ending request: urn:request:6d29539c-81c7-437f-867b-3bd9a5a4d95f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d29539c-81c7-437f-867b-3bd9a5a4d95f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:52,636 [243] [DEBUG] [app] Ending request: urn:request:8cdea7af-caf7-4c11-a3c3-9a3692a3f860 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8cdea7af-caf7-4c11-a3c3-9a3692a3f860', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:52,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:16:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:52,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:52,645 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:52,647 [239] [DEBUG] [app] Ending request: urn:request:7083b102-d89c-408c-994b-9d8baeb7a4b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7083b102-d89c-408c-994b-9d8baeb7a4b0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 06:16:52,650 [244] [DEBUG] [app] Ending request: urn:request:1c4d9df3-e832-4d1c-aa64-9e2af73fdd8c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c4d9df3-e832-4d1c-aa64-9e2af73fdd8c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:52,651 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:52,651 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.026) securityworker stdout | 2023-01-17 06:16:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:09.230038+00:00 (in 16.392209 seconds) securityworker stdout | 2023-01-17 06:16:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:22 UTC)" (scheduled at 2023-01-17 06:16:52.837365+00:00) securityworker stdout | 2023-01-17 06:16:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:22 UTC)" executed successfully builder stdout | 2023-01-17 06:16:57,024 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:57,033 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:57,033 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:57,039 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:16:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:16:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:27.226424+00:00 (in 29.999578 seconds) buildlogsarchiver stdout | 2023-01-17 06:16:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:27 UTC)" (scheduled at 2023-01-17 06:16:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:16:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 16, 57, 227175), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:16:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:16:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:16:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:08.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 06:16:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:08 UTC)" (scheduled at 2023-01-17 06:16:58.735066+00:00) notificationworker stdout | 2023-01-17 06:16:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 58, 735958), True, datetime.datetime(2023, 1, 17, 6, 16, 58, 735958), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:58,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:58,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:16:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:16:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:47.932490+00:00 (in 48.594608 seconds) namespacegcworker stdout | 2023-01-17 06:16:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:59 UTC)" (scheduled at 2023-01-17 06:16:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:16:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:16:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 59, 338265), True, datetime.datetime(2023, 1, 17, 6, 16, 59, 338265), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:16:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:16:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:16:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:17:02,752 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:17:03,076 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:17:03,248 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:17:04,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:17:04,230 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:17:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:17:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:39.030814+00:00 (in 34.292337 seconds) repositorygcworker stdout | 2023-01-17 06:17:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:22:04 UTC)" (scheduled at 2023-01-17 06:17:04.738062+00:00) repositorygcworker stdout | 2023-01-17 06:17:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 17, 4, 738780), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:17:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 17, 4, 738780), True, datetime.datetime(2023, 1, 17, 6, 17, 4, 738780), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:17:04,749 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 17, 4, 738780), True, datetime.datetime(2023, 1, 17, 6, 17, 4, 738780), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 17, 4, 738780), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:17:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:17:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:22:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:17:05,338 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:17:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:17:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:35.848675+00:00 (in 29.999562 seconds) gcworker stdout | 2023-01-17 06:17:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:35 UTC)" (scheduled at 2023-01-17 06:17:05.848675+00:00) gcworker stdout | 2023-01-17 06:17:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726625849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:17:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:17:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:35 UTC)" executed successfully builder stdout | 2023-01-17 06:17:06,136 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:17:06,206 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 06:17:06,567 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:17:06,574 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:17:06,577 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:17:06,580 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:17:07,049 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:07,058 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:07,058 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:07,063 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:17:07,540 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:17:07,549 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-web stdout | 2023-01-17 06:17:07,625 [244] [DEBUG] [app] Starting request: urn:request:6e997a62-c170-4c18-81fc-29deb8501db9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:07,626 [239] [DEBUG] [app] Starting request: urn:request:94594fc5-8e95-486e-9871-040fda52483b (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:07,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:07,630 [240] [DEBUG] [app] Starting request: urn:request:4ed2d4b2-dc3b-4eaf-a844-d4e841601e00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:07,631 [240] [DEBUG] [app] Ending request: urn:request:4ed2d4b2-dc3b-4eaf-a844-d4e841601e00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ed2d4b2-dc3b-4eaf-a844-d4e841601e00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:07,631 [237] [DEBUG] [app] Starting request: urn:request:b7ff8924-3637-4f76-8404-576243d0232b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:07,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:17:07,631 [237] [DEBUG] [app] Ending request: urn:request:b7ff8924-3637-4f76-8404-576243d0232b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b7ff8924-3637-4f76-8404-576243d0232b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:17:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:07,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:07,636 [243] [DEBUG] [app] Starting request: urn:request:ee9adb8f-d41e-4678-ab01-a934bd7e1fe8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:07,636 [244] [DEBUG] [app] Starting request: urn:request:15409167-cf2d-419a-a4fe-d552017ee18f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:07,636 [243] [DEBUG] [app] Ending request: urn:request:ee9adb8f-d41e-4678-ab01-a934bd7e1fe8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee9adb8f-d41e-4678-ab01-a934bd7e1fe8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:07,636 [244] [DEBUG] [app] Ending request: urn:request:15409167-cf2d-419a-a4fe-d552017ee18f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15409167-cf2d-419a-a4fe-d552017ee18f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:07,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:07,645 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:07,645 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:07,650 [244] [DEBUG] [app] Ending request: urn:request:6e997a62-c170-4c18-81fc-29deb8501db9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e997a62-c170-4c18-81fc-29deb8501db9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:07,650 [239] [DEBUG] [app] Ending request: urn:request:94594fc5-8e95-486e-9871-040fda52483b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94594fc5-8e95-486e-9871-040fda52483b', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:07,650 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:07,650 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:07,651 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.026) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.026 47 0.026) gunicorn-web stdout | 2023-01-17 06:17:07,651 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:17:08,372 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:17:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:18.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 06:17:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:18 UTC)" (scheduled at 2023-01-17 06:17:08.735066+00:00) notificationworker stdout | 2023-01-17 06:17:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 8, 735908), True, datetime.datetime(2023, 1, 17, 6, 17, 8, 735908), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:17:09,086 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:17:09,089 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:17:09,091 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:17:09,094 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:17:09,096 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:17:09,098 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:17:09,101 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:17:09,103 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:17:09,105 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:17:09,108 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:17:09,112 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:17:09,114 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:17:09,116 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:17:09,119 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:17:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:22.837365+00:00 (in 13.606788 seconds) securityworker stdout | 2023-01-17 06:17:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:39 UTC)" (scheduled at 2023-01-17 06:17:09.230038+00:00) securityworker stdout | 2023-01-17 06:17:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:17:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:17:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:17:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:17:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:17:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:17:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:17:09,246 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:17:09,257 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:17:10,346 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:17:10,993 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:17:11,805 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:17:11,834 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:17:11,875 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:17:11,894 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:17:17,073 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:17,082 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:17,082 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:17,088 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:28.735066+00:00 (in 9.999572 seconds) notificationworker stdout | 2023-01-17 06:17:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:28 UTC)" (scheduled at 2023-01-17 06:17:18.735066+00:00) notificationworker stdout | 2023-01-17 06:17:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 18, 735890), True, datetime.datetime(2023, 1, 17, 6, 17, 18, 735890), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:17:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:17:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:25.334138+00:00 (in 5.091449 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:17:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:20 UTC)" (scheduled at 2023-01-17 06:17:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:17:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:17:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 20, 243123), True, datetime.datetime(2023, 1, 17, 6, 17, 20, 243123), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:17:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:17:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:17:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:17:22,625 [239] [DEBUG] [app] Starting request: urn:request:d47d32de-0db1-4390-88e2-3933b77456f9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:22,625 [244] [DEBUG] [app] Starting request: urn:request:63d07a00-0ed0-4d72-a024-9dc090679cb8 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:22,630 [237] [DEBUG] [app] Starting request: urn:request:1918c063-6f14-49e9-a7dd-c86988f15fe3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:22,631 [237] [DEBUG] [app] Ending request: urn:request:1918c063-6f14-49e9-a7dd-c86988f15fe3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1918c063-6f14-49e9-a7dd-c86988f15fe3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:22,631 [234] [DEBUG] [app] Starting request: urn:request:cf5d8149-578e-49ce-9040-09031ecf6bb7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:22,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:17:22,631 [234] [DEBUG] [app] Ending request: urn:request:cf5d8149-578e-49ce-9040-09031ecf6bb7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf5d8149-578e-49ce-9040-09031ecf6bb7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:17:22,631 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:17:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:22,635 [241] [DEBUG] [app] Starting request: urn:request:e8569be9-a879-4fd8-84de-6fc1966585b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:22,636 [241] [DEBUG] [app] Ending request: urn:request:e8569be9-a879-4fd8-84de-6fc1966585b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e8569be9-a879-4fd8-84de-6fc1966585b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:22,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:22,637 [243] [DEBUG] [app] Starting request: urn:request:2a850920-6af0-45a7-bbf6-3f27f007cefb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:22,637 [243] [DEBUG] [app] Ending request: urn:request:2a850920-6af0-45a7-bbf6-3f27f007cefb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2a850920-6af0-45a7-bbf6-3f27f007cefb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:22,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:22,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:22,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:22,647 [239] [DEBUG] [app] Ending request: urn:request:d47d32de-0db1-4390-88e2-3933b77456f9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d47d32de-0db1-4390-88e2-3933b77456f9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:22,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:17:22,649 [244] [DEBUG] [app] Ending request: urn:request:63d07a00-0ed0-4d72-a024-9dc090679cb8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63d07a00-0ed0-4d72-a024-9dc090679cb8', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:22,649 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:22,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.025) securityworker stdout | 2023-01-17 06:17:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:39.230038+00:00 (in 16.392201 seconds) securityworker stdout | 2023-01-17 06:17:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:52 UTC)" (scheduled at 2023-01-17 06:17:22.837365+00:00) securityworker stdout | 2023-01-17 06:17:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:17:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:17:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:48.830274+00:00 (in 24.187986 seconds) exportactionlogsworker stdout | 2023-01-17 06:17:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:24 UTC)" (scheduled at 2023-01-17 06:17:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:17:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:17:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:17:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:17:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:20.242198+00:00 (in 54.907559 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:17:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:25 UTC)" (scheduled at 2023-01-17 06:17:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:17:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:17:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:25 UTC)" executed successfully builder stdout | 2023-01-17 06:17:27,098 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:27,107 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:27,107 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:27,113 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:17:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:17:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:57.226424+00:00 (in 29.999594 seconds) buildlogsarchiver stdout | 2023-01-17 06:17:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:57 UTC)" (scheduled at 2023-01-17 06:17:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:17:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 17, 27, 227175), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:17:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:17:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:17:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:57 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:38.735066+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 06:17:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:38 UTC)" (scheduled at 2023-01-17 06:17:28.735066+00:00) notificationworker stdout | 2023-01-17 06:17:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 28, 735941), True, datetime.datetime(2023, 1, 17, 6, 17, 28, 735941), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:17:32,770 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:17:33,083 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:17:33,283 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:17:34,114 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:17:34,240 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:17:35,374 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:17:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:17:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:05.848675+00:00 (in 29.999384 seconds) gcworker stdout | 2023-01-17 06:17:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:05 UTC)" (scheduled at 2023-01-17 06:17:35.848675+00:00) gcworker stdout | 2023-01-17 06:17:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726655849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:17:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:17:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:05 UTC)" executed successfully builder stdout | 2023-01-17 06:17:36,147 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:17:36,241 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:17:36,584 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} namespacegcworker stdout | 2023-01-17 06:17:36,586 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:17:36,589 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:17:36,591 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:17:37,123 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:37,131 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:37,131 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:37,137 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:17:37,571 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:17:37,574 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:17:37,625 [244] [DEBUG] [app] Starting request: urn:request:14271f83-cc2a-4aa2-9e49-20cc9f1d85a6 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:37,625 [239] [DEBUG] [app] Starting request: urn:request:6935f59a-01e4-42be-88ac-9496da092182 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:37,630 [240] [DEBUG] [app] Starting request: urn:request:1550d38d-87be-41d3-86ab-7c535b9eb125 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:37,630 [242] [DEBUG] [app] Starting request: urn:request:960fb270-85b9-4d39-97a6-15aef656981b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:37,631 [240] [DEBUG] [app] Ending request: urn:request:1550d38d-87be-41d3-86ab-7c535b9eb125 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1550d38d-87be-41d3-86ab-7c535b9eb125', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:37,631 [242] [DEBUG] [app] Ending request: urn:request:960fb270-85b9-4d39-97a6-15aef656981b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:960fb270-85b9-4d39-97a6-15aef656981b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:17:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:17:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:37,635 [243] [DEBUG] [app] Starting request: urn:request:f6972836-d850-4547-bcc5-d7b5b190f399 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:37,635 [243] [DEBUG] [app] Ending request: urn:request:f6972836-d850-4547-bcc5-d7b5b190f399 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6972836-d850-4547-bcc5-d7b5b190f399', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:37,636 [239] [DEBUG] [app] Starting request: urn:request:3f4c65fe-8d8b-4e0d-ad10-70791a51abda (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:37,636 [239] [DEBUG] [app] Ending request: urn:request:3f4c65fe-8d8b-4e0d-ad10-70791a51abda (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f4c65fe-8d8b-4e0d-ad10-70791a51abda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:37,647 [239] [DEBUG] [app] Ending request: urn:request:6935f59a-01e4-42be-88ac-9496da092182 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6935f59a-01e4-42be-88ac-9496da092182', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:37,648 [244] [DEBUG] [app] Ending request: urn:request:14271f83-cc2a-4aa2-9e49-20cc9f1d85a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14271f83-cc2a-4aa2-9e49-20cc9f1d85a6', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:17:37,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:17:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:17:38,408 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:17:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:43.935629+00:00 (in 5.200119 seconds) notificationworker stdout | 2023-01-17 06:17:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:48 UTC)" (scheduled at 2023-01-17 06:17:38.735066+00:00) notificationworker stdout | 2023-01-17 06:17:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 38, 735906), True, datetime.datetime(2023, 1, 17, 6, 17, 38, 735906), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:17:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:17:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:48.738458+00:00 (in 9.707209 seconds) repositorygcworker stdout | 2023-01-17 06:17:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:39 UTC)" (scheduled at 2023-01-17 06:17:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:17:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:17:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 39, 31708), True, datetime.datetime(2023, 1, 17, 6, 17, 39, 31708), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:17:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:17:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:17:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:17:39,095 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:17:39,098 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:17:39,100 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:17:39,103 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:17:39,106 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:17:39,109 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:17:39,111 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:17:39,113 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:17:39,117 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:17:39,120 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:17:39,122 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:17:39,125 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:17:39,127 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:17:39,129 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:17:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:52.837365+00:00 (in 13.606897 seconds) securityworker stdout | 2023-01-17 06:17:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:09 UTC)" (scheduled at 2023-01-17 06:17:39.230038+00:00) securityworker stdout | 2023-01-17 06:17:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:17:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:17:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:17:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:09 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:17:39,281 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:17:39,286 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:17:40,381 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:17:41,020 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:17:41,837 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:17:41,858 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:17:41,907 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:17:41,929 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:17:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:48.735066+00:00 (in 4.799014 seconds) notificationworker stdout | 2023-01-17 06:17:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:43 UTC)" (scheduled at 2023-01-17 06:17:43.935629+00:00) notificationworker stdout | 2023-01-17 06:17:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:17:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:43 UTC)" executed successfully builder stdout | 2023-01-17 06:17:47,147 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:47,155 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:47,155 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:47,162 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:17:47,933 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:17:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:59.337413+00:00 (in 11.404279 seconds) namespacegcworker stdout | 2023-01-17 06:17:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:47 UTC)" (scheduled at 2023-01-17 06:17:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:17:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:17:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:58.735066+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 06:17:48,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:58 UTC)" (scheduled at 2023-01-17 06:17:48.735066+00:00) notificationworker stdout | 2023-01-17 06:17:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 48, 736162), True, datetime.datetime(2023, 1, 17, 6, 17, 48, 736162), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:17:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:17:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:39.030814+00:00 (in 50.291897 seconds) repositorygcworker stdout | 2023-01-17 06:17:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:48 UTC)" (scheduled at 2023-01-17 06:17:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:17:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:17:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:17:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:17:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:24.641816+00:00 (in 35.811070 seconds) exportactionlogsworker stdout | 2023-01-17 06:17:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:48 UTC)" (scheduled at 2023-01-17 06:17:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:17:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:17:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 48, 831090), True, datetime.datetime(2023, 1, 17, 6, 17, 48, 831090), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:17:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:17:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:17:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:17:52,625 [244] [DEBUG] [app] Starting request: urn:request:39f75ade-0159-488c-9521-a27a4b77c0de (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:52,626 [239] [DEBUG] [app] Starting request: urn:request:c1da2158-5e6e-441a-a336-b6c0eacdbf17 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:17:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:52,631 [232] [DEBUG] [app] Starting request: urn:request:b51e03cf-f729-419d-b4f3-54f240f5c7e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:52,631 [242] [DEBUG] [app] Starting request: urn:request:4a72768b-9b76-4e76-9c34-4582ea00bdb4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:52,631 [232] [DEBUG] [app] Ending request: urn:request:b51e03cf-f729-419d-b4f3-54f240f5c7e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b51e03cf-f729-419d-b4f3-54f240f5c7e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:17:52,631 [242] [DEBUG] [app] Ending request: urn:request:4a72768b-9b76-4e76-9c34-4582ea00bdb4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a72768b-9b76-4e76-9c34-4582ea00bdb4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:17:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:17:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:52,635 [244] [DEBUG] [app] Starting request: urn:request:cae7682d-542e-4ba3-868d-ce30de348c56 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:52,636 [244] [DEBUG] [app] Ending request: urn:request:cae7682d-542e-4ba3-868d-ce30de348c56 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cae7682d-542e-4ba3-868d-ce30de348c56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:52,637 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:52,637 [243] [DEBUG] [app] Starting request: urn:request:faa01732-0677-4cae-b8ee-100bb4bf873e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:52,638 [243] [DEBUG] [app] Ending request: urn:request:faa01732-0677-4cae-b8ee-100bb4bf873e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:faa01732-0677-4cae-b8ee-100bb4bf873e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:52,638 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:52,638 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:52,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:52,646 [239] [DEBUG] [app] Ending request: urn:request:c1da2158-5e6e-441a-a336-b6c0eacdbf17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1da2158-5e6e-441a-a336-b6c0eacdbf17', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 06:17:52,648 [244] [DEBUG] [app] Ending request: urn:request:39f75ade-0159-488c-9521-a27a4b77c0de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39f75ade-0159-488c-9521-a27a4b77c0de', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:17:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:09.230038+00:00 (in 16.392230 seconds) securityworker stdout | 2023-01-17 06:17:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:22 UTC)" (scheduled at 2023-01-17 06:17:52.837365+00:00) securityworker stdout | 2023-01-17 06:17:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:22 UTC)" executed successfully builder stdout | 2023-01-17 06:17:57,172 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:57,181 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:57,181 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:57,187 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:17:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:17:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:27.226424+00:00 (in 29.999593 seconds) buildlogsarchiver stdout | 2023-01-17 06:17:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:27 UTC)" (scheduled at 2023-01-17 06:17:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:17:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 17, 57, 227151), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:17:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:17:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:17:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:27 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:08.735066+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 06:17:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:08 UTC)" (scheduled at 2023-01-17 06:17:58.735066+00:00) notificationworker stdout | 2023-01-17 06:17:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 58, 735920), True, datetime.datetime(2023, 1, 17, 6, 17, 58, 735920), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:17:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:17:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:47.932490+00:00 (in 48.594550 seconds) namespacegcworker stdout | 2023-01-17 06:17:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:59 UTC)" (scheduled at 2023-01-17 06:17:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:17:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:17:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 59, 338323), True, datetime.datetime(2023, 1, 17, 6, 17, 59, 338323), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:17:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:17:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:17:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:18:02,805 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:18:03,117 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:18:03,314 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:18:04,150 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:18:04,264 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:18:05,409 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:18:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:18:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:35.848675+00:00 (in 29.999540 seconds) gcworker stdout | 2023-01-17 06:18:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:35 UTC)" (scheduled at 2023-01-17 06:18:05.848675+00:00) gcworker stdout | 2023-01-17 06:18:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726685849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:18:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:18:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:35 UTC)" executed successfully builder stdout | 2023-01-17 06:18:06,162 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:18:06,254 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:18:06,592 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} namespacegcworker stdout | 2023-01-17 06:18:06,595 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-secscan stdout | 2023-01-17 06:18:06,597 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:18:06,600 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} builder stdout | 2023-01-17 06:18:07,197 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:07,206 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:07,206 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:07,212 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:18:07,585 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:18:07,599 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:18:07,625 [241] [DEBUG] [app] Starting request: urn:request:17d8f8a9-2047-4e24-9a90-2fff8725d19a (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:07,625 [239] [DEBUG] [app] Starting request: urn:request:8df6ec4b-0369-4b9f-bc04-8744212a86d9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:07,631 [242] [DEBUG] [app] Starting request: urn:request:bd7a6086-6ac7-4814-ad13-c398269f4445 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:07,631 [240] [DEBUG] [app] Starting request: urn:request:0d30d965-0fa5-45ec-ba67-8c96ea603393 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:07,631 [242] [DEBUG] [app] Ending request: urn:request:bd7a6086-6ac7-4814-ad13-c398269f4445 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd7a6086-6ac7-4814-ad13-c398269f4445', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:18:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:18:07,631 [240] [DEBUG] [app] Ending request: urn:request:0d30d965-0fa5-45ec-ba67-8c96ea603393 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d30d965-0fa5-45ec-ba67-8c96ea603393', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:18:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:18:07,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:18:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,634 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:07,636 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:07,636 [239] [DEBUG] [app] Starting request: urn:request:28811916-b02a-4455-90e6-5af8f466cca1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:07,636 [239] [DEBUG] [app] Ending request: urn:request:28811916-b02a-4455-90e6-5af8f466cca1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28811916-b02a-4455-90e6-5af8f466cca1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:07,637 [244] [DEBUG] [app] Starting request: urn:request:a35e1e33-4cac-40e1-a92e-1d8ea1e08c95 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:07,637 [244] [DEBUG] [app] Ending request: urn:request:a35e1e33-4cac-40e1-a92e-1d8ea1e08c95 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a35e1e33-4cac-40e1-a92e-1d8ea1e08c95', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:07,638 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:07,647 [239] [DEBUG] [app] Ending request: urn:request:8df6ec4b-0369-4b9f-bc04-8744212a86d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8df6ec4b-0369-4b9f-bc04-8744212a86d9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) gunicorn-web stdout | 2023-01-17 06:18:07,648 [241] [DEBUG] [app] Ending request: urn:request:17d8f8a9-2047-4e24-9a90-2fff8725d19a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:17d8f8a9-2047-4e24-9a90-2fff8725d19a', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:07,648 [241] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 06:18:07,649 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:18:08,431 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:18:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:18.735066+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 06:18:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:18 UTC)" (scheduled at 2023-01-17 06:18:08.735066+00:00) notificationworker stdout | 2023-01-17 06:18:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 8, 735908), True, datetime.datetime(2023, 1, 17, 6, 18, 8, 735908), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:18:09,104 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:18:09,107 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:18:09,109 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:18:09,111 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:18:09,115 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:18:09,117 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:18:09,122 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:18:09,125 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:18:09,128 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:18:09,130 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:18:09,132 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:18:09,135 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:18:09,137 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:18:09,140 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:18:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:22.837365+00:00 (in 13.606900 seconds) securityworker stdout | 2023-01-17 06:18:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:39 UTC)" (scheduled at 2023-01-17 06:18:09.230038+00:00) securityworker stdout | 2023-01-17 06:18:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:18:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:18:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:18:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:18:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:18:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:18:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:39 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:18:09,305 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:18:09,311 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} storagereplication stdout | 2023-01-17 06:18:10,412 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:18:11,055 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:18:11,853 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:18:11,874 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:18:11,934 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:18:11,947 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:18:17,220 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:17,230 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:17,230 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:17,236 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:28.735066+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 06:18:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:28 UTC)" (scheduled at 2023-01-17 06:18:18.735066+00:00) notificationworker stdout | 2023-01-17 06:18:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 18, 735919), True, datetime.datetime(2023, 1, 17, 6, 18, 18, 735919), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:18:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:18:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:25.334138+00:00 (in 5.091459 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:18:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:20 UTC)" (scheduled at 2023-01-17 06:18:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:18:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:18:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 20, 243091), True, datetime.datetime(2023, 1, 17, 6, 18, 20, 243091), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:18:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:18:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:18:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:22,625 [239] [DEBUG] [app] Starting request: urn:request:a62d9ac7-61aa-45f4-931b-56eae181ce25 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:22,626 [243] [DEBUG] [app] Starting request: urn:request:1e69a190-653b-45bc-8242-cdebd54f5561 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:22,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:22,630 [234] [DEBUG] [app] Starting request: urn:request:af6daf13-aece-4a8d-884e-16694aca7bac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:22,631 [242] [DEBUG] [app] Starting request: urn:request:78d12732-5644-4e3c-8143-251a324d8f95 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:22,631 [234] [DEBUG] [app] Ending request: urn:request:af6daf13-aece-4a8d-884e-16694aca7bac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af6daf13-aece-4a8d-884e-16694aca7bac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:18:22,631 [234] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:18:22,631 [242] [DEBUG] [app] Ending request: urn:request:78d12732-5644-4e3c-8143-251a324d8f95 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78d12732-5644-4e3c-8143-251a324d8f95', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:18:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:22,636 [244] [DEBUG] [app] Starting request: urn:request:c15cbe93-d909-4c05-8f9b-bf0e77bf249d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:22,636 [241] [DEBUG] [app] Starting request: urn:request:e2972357-6f11-40d5-b340-064d50a9212c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:22,636 [244] [DEBUG] [app] Ending request: urn:request:c15cbe93-d909-4c05-8f9b-bf0e77bf249d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c15cbe93-d909-4c05-8f9b-bf0e77bf249d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:22,636 [241] [DEBUG] [app] Ending request: urn:request:e2972357-6f11-40d5-b340-064d50a9212c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e2972357-6f11-40d5-b340-064d50a9212c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:22,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:22,648 [243] [DEBUG] [app] Ending request: urn:request:1e69a190-653b-45bc-8242-cdebd54f5561 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e69a190-653b-45bc-8242-cdebd54f5561', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:22,648 [239] [DEBUG] [app] Ending request: urn:request:a62d9ac7-61aa-45f4-931b-56eae181ce25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a62d9ac7-61aa-45f4-931b-56eae181ce25', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:22,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:18:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:18:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:18:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:39.230038+00:00 (in 16.392232 seconds) securityworker stdout | 2023-01-17 06:18:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:52 UTC)" (scheduled at 2023-01-17 06:18:22.837365+00:00) securityworker stdout | 2023-01-17 06:18:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:18:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:18:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:48.830274+00:00 (in 24.188023 seconds) exportactionlogsworker stdout | 2023-01-17 06:18:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:24 UTC)" (scheduled at 2023-01-17 06:18:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:18:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:18:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:18:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:18:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:20.242198+00:00 (in 54.907565 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:18:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:25 UTC)" (scheduled at 2023-01-17 06:18:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:18:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:18:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:18:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:18:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:57.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 06:18:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:57 UTC)" (scheduled at 2023-01-17 06:18:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:18:27,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 18, 27, 227243), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:18:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:18:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:18:27,236 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:57 UTC)" executed successfully builder stdout | 2023-01-17 06:18:27,245 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:27,252 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:27,252 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:27,258 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:38.735066+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 06:18:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:38 UTC)" (scheduled at 2023-01-17 06:18:28.735066+00:00) notificationworker stdout | 2023-01-17 06:18:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 28, 735993), True, datetime.datetime(2023, 1, 17, 6, 18, 28, 735993), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:18:32,841 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:18:33,153 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:18:33,348 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:18:34,186 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:18:34,272 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:18:35,416 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:18:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:18:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:05.848675+00:00 (in 29.999498 seconds) gcworker stdout | 2023-01-17 06:18:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:05 UTC)" (scheduled at 2023-01-17 06:18:35.848675+00:00) gcworker stdout | 2023-01-17 06:18:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726715849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:18:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:18:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:05 UTC)" executed successfully builder stdout | 2023-01-17 06:18:36,184 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:18:36,289 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:18:36,599 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:18:36,603 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:18:36,606 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:18:36,626 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:18:37,268 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:37,276 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:37,276 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:37,282 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:18:37,621 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} gunicorn-web stdout | 2023-01-17 06:18:37,625 [244] [DEBUG] [app] Starting request: urn:request:4722a385-e1b3-4fd7-ab1b-fed95f6dc271 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:37,625 [243] [DEBUG] [app] Starting request: urn:request:c4bc33b4-7d06-4d45-89c1-2ecf94aef9b4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:37,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:37,630 [242] [DEBUG] [app] Starting request: urn:request:b0ca3fde-17bf-40b9-9849-ba0cc0524d24 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:37,630 [242] [DEBUG] [app] Ending request: urn:request:b0ca3fde-17bf-40b9-9849-ba0cc0524d24 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0ca3fde-17bf-40b9-9849-ba0cc0524d24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:18:37,630 [232] [DEBUG] [app] Starting request: urn:request:d2db6446-ad4d-4fab-9a9b-8e8b730f175e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:18:37,631 [232] [DEBUG] [app] Ending request: urn:request:d2db6446-ad4d-4fab-9a9b-8e8b730f175e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2db6446-ad4d-4fab-9a9b-8e8b730f175e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:37,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:18:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:18:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:37,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost globalpromstats stdout | 2023-01-17 06:18:37,634 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 06:18:37,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:37,635 [239] [DEBUG] [app] Starting request: urn:request:f965db5f-64aa-4cdb-9e81-356815790d7f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:37,636 [243] [DEBUG] [app] Starting request: urn:request:8b9e18a6-16d9-4937-962a-13b24e6ac86e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:37,636 [239] [DEBUG] [app] Ending request: urn:request:f965db5f-64aa-4cdb-9e81-356815790d7f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f965db5f-64aa-4cdb-9e81-356815790d7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:37,636 [243] [DEBUG] [app] Ending request: urn:request:8b9e18a6-16d9-4937-962a-13b24e6ac86e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b9e18a6-16d9-4937-962a-13b24e6ac86e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:37,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:37,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:37,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:37,647 [243] [DEBUG] [app] Ending request: urn:request:c4bc33b4-7d06-4d45-89c1-2ecf94aef9b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4bc33b4-7d06-4d45-89c1-2ecf94aef9b4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:37,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:37,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:18:37,653 [244] [DEBUG] [app] Ending request: urn:request:4722a385-e1b3-4fd7-ab1b-fed95f6dc271 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4722a385-e1b3-4fd7-ab1b-fed95f6dc271', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:37,653 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:37,653 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) chunkcleanupworker stdout | 2023-01-17 06:18:38,467 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:18:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:43.935629+00:00 (in 5.200109 seconds) notificationworker stdout | 2023-01-17 06:18:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:48 UTC)" (scheduled at 2023-01-17 06:18:38.735066+00:00) notificationworker stdout | 2023-01-17 06:18:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 38, 735907), True, datetime.datetime(2023, 1, 17, 6, 18, 38, 735907), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:18:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:18:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:48.738458+00:00 (in 9.707211 seconds) repositorygcworker stdout | 2023-01-17 06:18:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:39 UTC)" (scheduled at 2023-01-17 06:18:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:18:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:18:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 39, 31666), True, datetime.datetime(2023, 1, 17, 6, 18, 39, 31666), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:18:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:18:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:18:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:18:39,112 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:18:39,114 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:18:39,117 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:18:39,121 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:18:39,124 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:18:39,127 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:18:39,132 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:18:39,135 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:18:39,137 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:18:39,139 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-web stdout | 2023-01-17 06:18:39,142 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:18:39,145 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:18:39,147 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:18:39,150 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:18:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:52.837365+00:00 (in 13.606859 seconds) securityworker stdout | 2023-01-17 06:18:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:09 UTC)" (scheduled at 2023-01-17 06:18:39.230038+00:00) securityworker stdout | 2023-01-17 06:18:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:18:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:18:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:18:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:18:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:18:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:18:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:09 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:18:39,327 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:18:39,335 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:18:40,425 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:18:41,091 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:18:41,877 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:18:41,908 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:18:41,945 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:18:41,980 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:18:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:48.735066+00:00 (in 4.798968 seconds) notificationworker stdout | 2023-01-17 06:18:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:43 UTC)" (scheduled at 2023-01-17 06:18:43.935629+00:00) notificationworker stdout | 2023-01-17 06:18:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:18:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:43 UTC)" executed successfully builder stdout | 2023-01-17 06:18:47,292 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:47,300 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:47,300 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:47,306 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:18:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:18:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:59.337413+00:00 (in 11.404478 seconds) namespacegcworker stdout | 2023-01-17 06:18:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:47 UTC)" (scheduled at 2023-01-17 06:18:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:18:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:18:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:18:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:56.935055+00:00 (in 8.199447 seconds) notificationworker stdout | 2023-01-17 06:18:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:58 UTC)" (scheduled at 2023-01-17 06:18:48.735066+00:00) notificationworker stdout | 2023-01-17 06:18:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 48, 735988), True, datetime.datetime(2023, 1, 17, 6, 18, 48, 735988), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:18:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:18:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:39.030814+00:00 (in 50.291870 seconds) repositorygcworker stdout | 2023-01-17 06:18:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:48 UTC)" (scheduled at 2023-01-17 06:18:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:18:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:18:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:18:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:18:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:18:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:24.641816+00:00 (in 35.811121 seconds) exportactionlogsworker stdout | 2023-01-17 06:18:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:48 UTC)" (scheduled at 2023-01-17 06:18:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:18:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:18:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 48, 831067), True, datetime.datetime(2023, 1, 17, 6, 18, 48, 831067), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:18:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:18:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:18:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:52,625 [239] [DEBUG] [app] Starting request: urn:request:0c5999b7-76ad-4fab-b32f-ab11b868ff56 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:52,625 [243] [DEBUG] [app] Starting request: urn:request:05db9850-97e1-4b0d-88eb-2503c74ac9aa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:18:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:52,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:52,630 [242] [DEBUG] [app] Starting request: urn:request:91d68609-10c8-48f9-9fe0-504c60dce3c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:52,630 [240] [DEBUG] [app] Starting request: urn:request:1d97c4bc-b505-4869-9df6-8f9ba7a68583 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:52,630 [242] [DEBUG] [app] Ending request: urn:request:91d68609-10c8-48f9-9fe0-504c60dce3c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:91d68609-10c8-48f9-9fe0-504c60dce3c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:18:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:18:52,631 [240] [DEBUG] [app] Ending request: urn:request:1d97c4bc-b505-4869-9df6-8f9ba7a68583 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d97c4bc-b505-4869-9df6-8f9ba7a68583', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:18:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:52,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:52,635 [244] [DEBUG] [app] Starting request: urn:request:0300c866-55d6-4e42-aaca-4482845d8eec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:52,635 [241] [DEBUG] [app] Starting request: urn:request:59aaec6a-5df4-4884-94cb-55d09a99e22d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:52,636 [244] [DEBUG] [app] Ending request: urn:request:0300c866-55d6-4e42-aaca-4482845d8eec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0300c866-55d6-4e42-aaca-4482845d8eec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:52,636 [241] [DEBUG] [app] Ending request: urn:request:59aaec6a-5df4-4884-94cb-55d09a99e22d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59aaec6a-5df4-4884-94cb-55d09a99e22d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:52,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:52,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:52,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:18:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:52,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:52,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:52,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:52,646 [243] [DEBUG] [app] Ending request: urn:request:05db9850-97e1-4b0d-88eb-2503c74ac9aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05db9850-97e1-4b0d-88eb-2503c74ac9aa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:52,647 [239] [DEBUG] [app] Ending request: urn:request:0c5999b7-76ad-4fab-b32f-ab11b868ff56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c5999b7-76ad-4fab-b32f-ab11b868ff56', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:52,647 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:52,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:52,647 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:18:52,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 06:18:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:09.230038+00:00 (in 16.392193 seconds) securityworker stdout | 2023-01-17 06:18:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:22 UTC)" (scheduled at 2023-01-17 06:18:52.837365+00:00) securityworker stdout | 2023-01-17 06:18:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:52,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:52,849 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:18:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:58.735066+00:00 (in 1.799571 seconds) notificationworker stdout | 2023-01-17 06:18:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:23:56 UTC)" (scheduled at 2023-01-17 06:18:56.935055+00:00) notificationworker stdout | 2023-01-17 06:18:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 18, 56, 935827), 'notification/%']) notificationworker stdout | 2023-01-17 06:18:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 18, 56, 935827), True, datetime.datetime(2023, 1, 17, 6, 18, 56, 935827), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:18:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 18, 56, 935827), True, datetime.datetime(2023, 1, 17, 6, 18, 56, 935827), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 18, 56, 935827), 'notification/%']) notificationworker stdout | 2023-01-17 06:18:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:23:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:18:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:18:57,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:27.226424+00:00 (in 29.999574 seconds) buildlogsarchiver stdout | 2023-01-17 06:18:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:27 UTC)" (scheduled at 2023-01-17 06:18:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:18:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 18, 57, 227185), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:18:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:18:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:18:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:27 UTC)" executed successfully builder stdout | 2023-01-17 06:18:57,316 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:57,324 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:57,324 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:57,330 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:08.735066+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 06:18:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:08 UTC)" (scheduled at 2023-01-17 06:18:58.735066+00:00) notificationworker stdout | 2023-01-17 06:18:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 58, 735954), True, datetime.datetime(2023, 1, 17, 6, 18, 58, 735954), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:18:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:18:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:15.931994+00:00 (in 16.594110 seconds) namespacegcworker stdout | 2023-01-17 06:18:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:59 UTC)" (scheduled at 2023-01-17 06:18:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:18:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:18:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 59, 338291), True, datetime.datetime(2023, 1, 17, 6, 18, 59, 338291), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:18:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:18:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:18:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:19:02,856 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:19:03,177 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:19:03,382 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:19:04,222 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:19:04,290 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:19:05,423 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:19:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:19:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:35.848675+00:00 (in 29.999457 seconds) gcworker stdout | 2023-01-17 06:19:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:35 UTC)" (scheduled at 2023-01-17 06:19:05.848675+00:00) gcworker stdout | 2023-01-17 06:19:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726745849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:19:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:19:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:19:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:35 UTC)" executed successfully builder stdout | 2023-01-17 06:19:06,207 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:19:06,324 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:19:06,606 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:19:06,609 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:19:06,612 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:19:06,632 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:19:07,332 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:07,340 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:07,340 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:07,346 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,626 [243] [DEBUG] [app] Starting request: urn:request:4df0fe91-7ded-4645-be2a-4ca098c2b7a7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:07,626 [241] [DEBUG] [app] Starting request: urn:request:56c546f9-58da-4822-98ed-5d85a91c8b2e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,627 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:07,630 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:07,631 [232] [DEBUG] [app] Starting request: urn:request:23ce9270-c541-4df1-9ef2-3c19e6522c40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:07,631 [237] [DEBUG] [app] Starting request: urn:request:f53e21c6-bef1-40fe-9637-c7fc75c3aa45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:07,631 [237] [DEBUG] [app] Ending request: urn:request:f53e21c6-bef1-40fe-9637-c7fc75c3aa45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f53e21c6-bef1-40fe-9637-c7fc75c3aa45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:07,631 [232] [DEBUG] [app] Ending request: urn:request:23ce9270-c541-4df1-9ef2-3c19e6522c40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23ce9270-c541-4df1-9ef2-3c19e6522c40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:07,632 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:19:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:07,632 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:19:07,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:07,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:07,636 [239] [DEBUG] [app] Starting request: urn:request:cb27b00f-9314-43f9-befb-f478720062a1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:07,637 [244] [DEBUG] [app] Starting request: urn:request:0203f719-fac5-46bf-976c-32b721c9f39e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:07,637 [239] [DEBUG] [app] Ending request: urn:request:cb27b00f-9314-43f9-befb-f478720062a1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cb27b00f-9314-43f9-befb-f478720062a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:07,637 [244] [DEBUG] [app] Ending request: urn:request:0203f719-fac5-46bf-976c-32b721c9f39e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0203f719-fac5-46bf-976c-32b721c9f39e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:07,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:07,638 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:07,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:07,648 [241] [DEBUG] [app] Ending request: urn:request:56c546f9-58da-4822-98ed-5d85a91c8b2e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:56c546f9-58da-4822-98ed-5d85a91c8b2e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:07,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:19:07,648 [243] [DEBUG] [app] Ending request: urn:request:4df0fe91-7ded-4645-be2a-4ca098c2b7a7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4df0fe91-7ded-4645-be2a-4ca098c2b7a7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) repositoryactioncounter stdout | 2023-01-17 06:19:07,657 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:19:07,659 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:19:08,478 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:19:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:18.735066+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 06:19:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:18 UTC)" (scheduled at 2023-01-17 06:19:08.735066+00:00) notificationworker stdout | 2023-01-17 06:19:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 8, 735913), True, datetime.datetime(2023, 1, 17, 6, 19, 8, 735913), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:19:09,119 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:19:09,121 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:19:09,126 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-web stdout | 2023-01-17 06:19:09,130 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:19:09,132 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:19:09,136 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:19:09,140 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:19:09,143 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:19:09,146 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:19:09,150 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:19:09,153 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:19:09,155 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:19:09,157 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:19:09,159 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:19:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:22.837365+00:00 (in 13.606886 seconds) securityworker stdout | 2023-01-17 06:19:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:39 UTC)" (scheduled at 2023-01-17 06:19:09.230038+00:00) securityworker stdout | 2023-01-17 06:19:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:19:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:19:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:09,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:09,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:19:09,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:19:09,338 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:19:09,370 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:19:10,432 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:19:11,115 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:19:11,893 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:19:11,914 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:19:11,967 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:19:12,006 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} namespacegcworker stdout | 2023-01-17 06:19:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:19:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:47.932490+00:00 (in 32.000006 seconds) namespacegcworker stdout | 2023-01-17 06:19:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:24:15 UTC)" (scheduled at 2023-01-17 06:19:15.931994+00:00) namespacegcworker stdout | 2023-01-17 06:19:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 19, 15, 932905), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:19:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 19, 15, 932905), True, datetime.datetime(2023, 1, 17, 6, 19, 15, 932905), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:19:15,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 19, 15, 932905), True, datetime.datetime(2023, 1, 17, 6, 19, 15, 932905), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 19, 15, 932905), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:19:15,944 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:19:15,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:24:15 UTC)" executed successfully builder stdout | 2023-01-17 06:19:17,356 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:17,365 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:17,365 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:17,371 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:28.735066+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 06:19:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:28 UTC)" (scheduled at 2023-01-17 06:19:18.735066+00:00) notificationworker stdout | 2023-01-17 06:19:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 18, 735910), True, datetime.datetime(2023, 1, 17, 6, 19, 18, 735910), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:19:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:19:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:25.334138+00:00 (in 5.091494 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:19:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:20 UTC)" (scheduled at 2023-01-17 06:19:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:19:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:19:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 20, 243055), True, datetime.datetime(2023, 1, 17, 6, 19, 20, 243055), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:19:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:19:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:19:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:22,626 [239] [DEBUG] [app] Starting request: urn:request:85df8d03-ffb4-45b8-953c-68f4d1e6f0df (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:22,626 [244] [DEBUG] [app] Starting request: urn:request:9e9c79bc-fe1f-4f42-b403-90b6e5fae5ee (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:22,631 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:22,631 [240] [DEBUG] [app] Starting request: urn:request:3a788fb0-21a9-4d57-b575-154b4f02f534 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:22,631 [242] [DEBUG] [app] Starting request: urn:request:d1ff0894-dfbe-4811-9a78-377eb5d9a855 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:22,632 [240] [DEBUG] [app] Ending request: urn:request:3a788fb0-21a9-4d57-b575-154b4f02f534 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a788fb0-21a9-4d57-b575-154b4f02f534', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:19:22,632 [242] [DEBUG] [app] Ending request: urn:request:d1ff0894-dfbe-4811-9a78-377eb5d9a855 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d1ff0894-dfbe-4811-9a78-377eb5d9a855', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:19:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:22,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:22,636 [243] [DEBUG] [app] Starting request: urn:request:b2c98cf0-4242-4dcf-a25a-4fed9e7ef81f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:22,637 [241] [DEBUG] [app] Starting request: urn:request:fbccd17c-1383-4071-a76a-9f9ffaf9bdcf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:22,637 [243] [DEBUG] [app] Ending request: urn:request:b2c98cf0-4242-4dcf-a25a-4fed9e7ef81f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2c98cf0-4242-4dcf-a25a-4fed9e7ef81f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:19:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:22,637 [241] [DEBUG] [app] Ending request: urn:request:fbccd17c-1383-4071-a76a-9f9ffaf9bdcf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fbccd17c-1383-4071-a76a-9f9ffaf9bdcf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:22,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:22,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:22,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:22,648 [244] [DEBUG] [app] Ending request: urn:request:9e9c79bc-fe1f-4f42-b403-90b6e5fae5ee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e9c79bc-fe1f-4f42-b403-90b6e5fae5ee', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:22,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:22,648 [239] [DEBUG] [app] Ending request: urn:request:85df8d03-ffb4-45b8-953c-68f4d1e6f0df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:85df8d03-ffb4-45b8-953c-68f4d1e6f0df', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:22,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:19:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:19:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:19:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:39.230038+00:00 (in 16.392251 seconds) securityworker stdout | 2023-01-17 06:19:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:52 UTC)" (scheduled at 2023-01-17 06:19:22.837365+00:00) securityworker stdout | 2023-01-17 06:19:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:19:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:19:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:48.830274+00:00 (in 24.188031 seconds) exportactionlogsworker stdout | 2023-01-17 06:19:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:24 UTC)" (scheduled at 2023-01-17 06:19:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:19:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:19:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:19:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:19:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:20.242198+00:00 (in 54.907575 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:19:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:25 UTC)" (scheduled at 2023-01-17 06:19:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:19:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:19:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:19:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:19:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:57.226424+00:00 (in 29.999577 seconds) buildlogsarchiver stdout | 2023-01-17 06:19:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:57 UTC)" (scheduled at 2023-01-17 06:19:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:19:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 19, 27, 227228), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:19:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:19:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:19:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:57 UTC)" executed successfully builder stdout | 2023-01-17 06:19:27,374 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:27,384 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:27,384 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:27,390 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:38.735066+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 06:19:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:38 UTC)" (scheduled at 2023-01-17 06:19:28.735066+00:00) notificationworker stdout | 2023-01-17 06:19:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 28, 735903), True, datetime.datetime(2023, 1, 17, 6, 19, 28, 735903), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:19:32,877 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:19:33,205 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:19:33,388 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:19:34,253 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:19:34,307 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:19:35,445 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:19:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:19:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:05.848675+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 06:19:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:05 UTC)" (scheduled at 2023-01-17 06:19:35.848675+00:00) gcworker stdout | 2023-01-17 06:19:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726775849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:19:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:19:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:19:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:05 UTC)" executed successfully builder stdout | 2023-01-17 06:19:36,229 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:19:36,359 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:19:36,614 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:19:36,616 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:19:36,619 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:19:36,652 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:19:37,400 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:37,408 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:37,408 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:37,414 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:37,625 [244] [DEBUG] [app] Starting request: urn:request:af398074-1c2c-40e3-803e-4f112e8c723c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:37,625 [239] [DEBUG] [app] Starting request: urn:request:b4980ca6-660f-49c4-980c-fb1be71ff9bd (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:37,630 [240] [DEBUG] [app] Starting request: urn:request:a5ee55e0-a689-478e-bdf9-fa150f379a08 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:37,631 [242] [DEBUG] [app] Starting request: urn:request:97c1295b-7f8f-4f5b-8a69-fa85fc5fc2df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:37,631 [240] [DEBUG] [app] Ending request: urn:request:a5ee55e0-a689-478e-bdf9-fa150f379a08 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5ee55e0-a689-478e-bdf9-fa150f379a08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:37,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:19:37,631 [242] [DEBUG] [app] Ending request: urn:request:97c1295b-7f8f-4f5b-8a69-fa85fc5fc2df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97c1295b-7f8f-4f5b-8a69-fa85fc5fc2df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:37,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:19:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:37,635 [239] [DEBUG] [app] Starting request: urn:request:9ed37092-9bf8-4498-9c6e-27021b48d9fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:37,635 [239] [DEBUG] [app] Ending request: urn:request:9ed37092-9bf8-4498-9c6e-27021b48d9fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ed37092-9bf8-4498-9c6e-27021b48d9fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:37,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:37,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:37,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:37,637 [241] [DEBUG] [app] Starting request: urn:request:20fd9a33-2053-4ca6-a698-293eca2fb10d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:37,638 [241] [DEBUG] [app] Ending request: urn:request:20fd9a33-2053-4ca6-a698-293eca2fb10d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:20fd9a33-2053-4ca6-a698-293eca2fb10d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:37,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:37,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:37,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:37,646 [244] [DEBUG] [app] Ending request: urn:request:af398074-1c2c-40e3-803e-4f112e8c723c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af398074-1c2c-40e3-803e-4f112e8c723c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:37,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 06:19:37,648 [239] [DEBUG] [app] Ending request: urn:request:b4980ca6-660f-49c4-980c-fb1be71ff9bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b4980ca6-660f-49c4-980c-fb1be71ff9bd', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:37,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:19:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 06:19:37,685 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:19:37,687 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 06:19:38,503 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:19:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:43.935629+00:00 (in 5.200142 seconds) notificationworker stdout | 2023-01-17 06:19:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:48 UTC)" (scheduled at 2023-01-17 06:19:38.735066+00:00) notificationworker stdout | 2023-01-17 06:19:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 38, 735865), True, datetime.datetime(2023, 1, 17, 6, 19, 38, 735865), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:19:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:19:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:48.738458+00:00 (in 9.707200 seconds) repositorygcworker stdout | 2023-01-17 06:19:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:39 UTC)" (scheduled at 2023-01-17 06:19:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:19:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:19:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 39, 31642), True, datetime.datetime(2023, 1, 17, 6, 19, 39, 31642), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:19:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:19:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:19:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:19:39,127 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:19:39,132 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:19:39,135 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-web stdout | 2023-01-17 06:19:39,137 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:19:39,141 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:19:39,143 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:19:39,147 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:19:39,152 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:19:39,155 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:19:39,158 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:19:39,160 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:19:39,163 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:19:39,165 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:19:39,167 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:19:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:52.837365+00:00 (in 13.606888 seconds) securityworker stdout | 2023-01-17 06:19:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:09 UTC)" (scheduled at 2023-01-17 06:19:39.230038+00:00) securityworker stdout | 2023-01-17 06:19:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:19:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:19:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:19:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:19:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:19:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:19:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:09 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:19:39,359 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:19:39,405 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:19:40,468 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:19:41,144 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:19:41,929 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:19:41,931 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:19:41,982 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:19:42,033 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:19:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:48.735066+00:00 (in 4.798991 seconds) notificationworker stdout | 2023-01-17 06:19:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:43 UTC)" (scheduled at 2023-01-17 06:19:43.935629+00:00) notificationworker stdout | 2023-01-17 06:19:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:19:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:43 UTC)" executed successfully builder stdout | 2023-01-17 06:19:47,424 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:47,434 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:47,434 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:47,439 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:19:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:19:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:59.337413+00:00 (in 11.404407 seconds) namespacegcworker stdout | 2023-01-17 06:19:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:47 UTC)" (scheduled at 2023-01-17 06:19:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:19:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:19:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:19:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:58.735066+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 06:19:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:58 UTC)" (scheduled at 2023-01-17 06:19:48.735066+00:00) notificationworker stdout | 2023-01-17 06:19:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 48, 735916), True, datetime.datetime(2023, 1, 17, 6, 19, 48, 735916), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:19:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:19:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:39.030814+00:00 (in 50.291890 seconds) repositorygcworker stdout | 2023-01-17 06:19:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:48 UTC)" (scheduled at 2023-01-17 06:19:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:19:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:19:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:19:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:19:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:19:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:24.641816+00:00 (in 35.811057 seconds) exportactionlogsworker stdout | 2023-01-17 06:19:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:48 UTC)" (scheduled at 2023-01-17 06:19:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:19:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:19:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 48, 831120), True, datetime.datetime(2023, 1, 17, 6, 19, 48, 831120), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:19:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:19:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:19:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:52,625 [244] [DEBUG] [app] Starting request: urn:request:d338b0a9-d486-4204-94ad-1a78d068b3b2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:52,625 [239] [DEBUG] [app] Starting request: urn:request:81682df9-0130-429c-83f2-1a2efcd683c3 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:19:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:52,630 [240] [DEBUG] [app] Starting request: urn:request:8f6e37ab-5e6b-4268-afea-f150b360ae81 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:52,631 [237] [DEBUG] [app] Starting request: urn:request:132c6ab7-772d-4298-be02-7262b04a1c9b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:52,631 [240] [DEBUG] [app] Ending request: urn:request:8f6e37ab-5e6b-4268-afea-f150b360ae81 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8f6e37ab-5e6b-4268-afea-f150b360ae81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:19:52,631 [237] [DEBUG] [app] Ending request: urn:request:132c6ab7-772d-4298-be02-7262b04a1c9b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:132c6ab7-772d-4298-be02-7262b04a1c9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:52,631 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:19:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:52,636 [241] [DEBUG] [app] Starting request: urn:request:05a8e3b9-b432-493a-a6eb-9905f9f6e045 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:52,636 [239] [DEBUG] [app] Starting request: urn:request:212da179-4aeb-48f8-8b8f-40ba82999c7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:52,636 [241] [DEBUG] [app] Ending request: urn:request:05a8e3b9-b432-493a-a6eb-9905f9f6e045 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05a8e3b9-b432-493a-a6eb-9905f9f6e045', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:52,637 [239] [DEBUG] [app] Ending request: urn:request:212da179-4aeb-48f8-8b8f-40ba82999c7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:212da179-4aeb-48f8-8b8f-40ba82999c7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:52,648 [244] [DEBUG] [app] Ending request: urn:request:d338b0a9-d486-4204-94ad-1a78d068b3b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d338b0a9-d486-4204-94ad-1a78d068b3b2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:52,648 [239] [DEBUG] [app] Ending request: urn:request:81682df9-0130-429c-83f2-1a2efcd683c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81682df9-0130-429c-83f2-1a2efcd683c3', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:52,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:19:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:19:52,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:19:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:09.230038+00:00 (in 16.392181 seconds) securityworker stdout | 2023-01-17 06:19:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:22 UTC)" (scheduled at 2023-01-17 06:19:52.837365+00:00) securityworker stdout | 2023-01-17 06:19:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:19:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:19:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:27.226424+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 06:19:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:27 UTC)" (scheduled at 2023-01-17 06:19:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:19:57,228 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 19, 57, 227296), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:19:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:19:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:19:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:27 UTC)" executed successfully builder stdout | 2023-01-17 06:19:57,449 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:57,458 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:57,458 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:57,463 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:08.735066+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 06:19:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:08 UTC)" (scheduled at 2023-01-17 06:19:58.735066+00:00) notificationworker stdout | 2023-01-17 06:19:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 58, 735961), True, datetime.datetime(2023, 1, 17, 6, 19, 58, 735961), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:58,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:58,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:58,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:19:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:19:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:47.932490+00:00 (in 48.594656 seconds) namespacegcworker stdout | 2023-01-17 06:19:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:59 UTC)" (scheduled at 2023-01-17 06:19:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:19:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:19:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 59, 338218), True, datetime.datetime(2023, 1, 17, 6, 19, 59, 338218), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:19:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:19:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:19:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:20:02,912 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:20:03,221 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:20:03,397 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:20:04,268 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:20:04,313 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:20:05,466 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:20:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:20:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:35.848675+00:00 (in 29.999512 seconds) gcworker stdout | 2023-01-17 06:20:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:35 UTC)" (scheduled at 2023-01-17 06:20:05.848675+00:00) gcworker stdout | 2023-01-17 06:20:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726805849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:20:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:20:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:35 UTC)" executed successfully builder stdout | 2023-01-17 06:20:06,251 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:20:06,394 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:20:06,621 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:20:06,623 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:20:06,625 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:20:06,674 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:20:07,473 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:07,482 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:07,482 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:07,488 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:07,626 [243] [DEBUG] [app] Starting request: urn:request:c8c9594a-b109-409f-a033-093482650473 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:07,626 [239] [DEBUG] [app] Starting request: urn:request:6a13fc95-e239-4136-a366-036e8cd675f1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:07,631 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:07,631 [232] [DEBUG] [app] Starting request: urn:request:eb887cbe-6c31-42ee-b010-1e6a68204596 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:07,632 [242] [DEBUG] [app] Starting request: urn:request:212c0f4b-c3a8-4817-9c80-0c5b57482f23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:07,632 [232] [DEBUG] [app] Ending request: urn:request:eb887cbe-6c31-42ee-b010-1e6a68204596 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb887cbe-6c31-42ee-b010-1e6a68204596', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:20:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:20:07,632 [242] [DEBUG] [app] Ending request: urn:request:212c0f4b-c3a8-4817-9c80-0c5b57482f23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:212c0f4b-c3a8-4817-9c80-0c5b57482f23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:20:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:07,633 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:07,634 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:07,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:07,636 [244] [DEBUG] [app] Starting request: urn:request:add2ed4d-6dea-434c-9692-8401f4ac96bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:07,637 [241] [DEBUG] [app] Starting request: urn:request:329cdb53-3ec8-42f1-9025-127714298030 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:07,637 [244] [DEBUG] [app] Ending request: urn:request:add2ed4d-6dea-434c-9692-8401f4ac96bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:add2ed4d-6dea-434c-9692-8401f4ac96bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:07,637 [241] [DEBUG] [app] Ending request: urn:request:329cdb53-3ec8-42f1-9025-127714298030 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:329cdb53-3ec8-42f1-9025-127714298030', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:07,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:07,638 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:07,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:07,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:07,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:07,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:07,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:07,648 [239] [DEBUG] [app] Ending request: urn:request:6a13fc95-e239-4136-a366-036e8cd675f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a13fc95-e239-4136-a366-036e8cd675f1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:07,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:20:07,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:07,648 [243] [DEBUG] [app] Ending request: urn:request:c8c9594a-b109-409f-a033-093482650473 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8c9594a-b109-409f-a033-093482650473', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:07,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:20:07,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 06:20:07,714 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:20:07,722 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 06:20:08,539 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:20:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:18.735066+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 06:20:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:18 UTC)" (scheduled at 2023-01-17 06:20:08.735066+00:00) notificationworker stdout | 2023-01-17 06:20:08,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 8, 736035), True, datetime.datetime(2023, 1, 17, 6, 20, 8, 736035), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:18 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:20:09,140 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:20:09,143 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:20:09,145 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:20:09,147 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:20:09,150 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:20:09,153 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:20:09,155 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:20:09,157 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:20:09,164 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:20:09,167 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:20:09,169 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:20:09,171 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:20:09,174 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:20:09,177 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:20:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:22.837365+00:00 (in 13.606851 seconds) securityworker stdout | 2023-01-17 06:20:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:39 UTC)" (scheduled at 2023-01-17 06:20:09.230038+00:00) securityworker stdout | 2023-01-17 06:20:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:20:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:20:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:20:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:20:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:20:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:20:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:20:09,395 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:20:09,433 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:20:10,483 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:20:11,180 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:20:11,965 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} expiredappspecifictokenworker stdout | 2023-01-17 06:20:11,968 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:20:11,998 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:20:12,054 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:20:17,498 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:17,507 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:17,507 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:17,513 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:28.735066+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 06:20:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:28 UTC)" (scheduled at 2023-01-17 06:20:18.735066+00:00) notificationworker stdout | 2023-01-17 06:20:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 18, 735973), True, datetime.datetime(2023, 1, 17, 6, 20, 18, 735973), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:20:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:20:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:25.334138+00:00 (in 5.091492 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:20:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:20 UTC)" (scheduled at 2023-01-17 06:20:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:20:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:20:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 20, 243041), True, datetime.datetime(2023, 1, 17, 6, 20, 20, 243041), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:20:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:20:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:20:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:20:22,626 [243] [DEBUG] [app] Starting request: urn:request:118c22e9-feee-45c3-82ab-fee4844f3dfa (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:22,626 [244] [DEBUG] [app] Starting request: urn:request:e5c2a60e-c0f8-45f0-bc5a-c6166d5f1eba (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:22,631 [240] [DEBUG] [app] Starting request: urn:request:30392603-64fa-4069-b1df-79e3026e181c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:22,631 [232] [DEBUG] [app] Starting request: urn:request:eb8040b3-f1a6-415f-92e9-1ca56bf01c46 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:22,631 [240] [DEBUG] [app] Ending request: urn:request:30392603-64fa-4069-b1df-79e3026e181c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30392603-64fa-4069-b1df-79e3026e181c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:20:22,631 [232] [DEBUG] [app] Ending request: urn:request:eb8040b3-f1a6-415f-92e9-1ca56bf01c46 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb8040b3-f1a6-415f-92e9-1ca56bf01c46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:20:22,632 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:20:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:20:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:22,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:22,636 [239] [DEBUG] [app] Starting request: urn:request:08c8943b-c831-4d7b-8c6c-1f8c73da0dfb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:22,636 [239] [DEBUG] [app] Ending request: urn:request:08c8943b-c831-4d7b-8c6c-1f8c73da0dfb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:08c8943b-c831-4d7b-8c6c-1f8c73da0dfb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:22,636 [243] [DEBUG] [app] Starting request: urn:request:fd476fd0-64b4-4efa-9382-10dafa2ad639 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:22,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:22,637 [243] [DEBUG] [app] Ending request: urn:request:fd476fd0-64b4-4efa-9382-10dafa2ad639 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd476fd0-64b4-4efa-9382-10dafa2ad639', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:22,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:22,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:22,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:22,648 [243] [DEBUG] [app] Ending request: urn:request:118c22e9-feee-45c3-82ab-fee4844f3dfa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:118c22e9-feee-45c3-82ab-fee4844f3dfa', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:22,648 [244] [DEBUG] [app] Ending request: urn:request:e5c2a60e-c0f8-45f0-bc5a-c6166d5f1eba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5c2a60e-c0f8-45f0-bc5a-c6166d5f1eba', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:22,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:20:22,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) securityworker stdout | 2023-01-17 06:20:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:39.230038+00:00 (in 16.392232 seconds) securityworker stdout | 2023-01-17 06:20:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:52 UTC)" (scheduled at 2023-01-17 06:20:22.837365+00:00) securityworker stdout | 2023-01-17 06:20:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:22,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:22,849 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:20:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:20:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:48.830274+00:00 (in 24.188000 seconds) exportactionlogsworker stdout | 2023-01-17 06:20:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:24 UTC)" (scheduled at 2023-01-17 06:20:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:20:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:20:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:20:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:20:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:20.242198+00:00 (in 54.907594 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:20:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:25 UTC)" (scheduled at 2023-01-17 06:20:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:20:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:20:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:20:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:20:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:57.226424+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 06:20:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:57 UTC)" (scheduled at 2023-01-17 06:20:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:20:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 20, 27, 227226), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:20:27,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:20:27,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:20:27,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:57 UTC)" executed successfully builder stdout | 2023-01-17 06:20:27,523 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:27,531 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:27,531 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:27,537 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:38.735066+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 06:20:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:38 UTC)" (scheduled at 2023-01-17 06:20:28.735066+00:00) notificationworker stdout | 2023-01-17 06:20:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 28, 735881), True, datetime.datetime(2023, 1, 17, 6, 20, 28, 735881), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:20:32,925 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:20:33,235 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:20:33,431 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:20:34,290 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:20:34,349 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:20:35,502 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:20:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:20:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:05.848675+00:00 (in 29.999523 seconds) gcworker stdout | 2023-01-17 06:20:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:05 UTC)" (scheduled at 2023-01-17 06:20:35.848675+00:00) gcworker stdout | 2023-01-17 06:20:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726835849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:20:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:20:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:35,859 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:05 UTC)" executed successfully builder stdout | 2023-01-17 06:20:36,281 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:20:36,405 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:20:36,627 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:20:36,629 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:20:36,632 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:20:36,682 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:20:37,547 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:37,556 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:37,556 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:37,562 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:37,625 [241] [DEBUG] [app] Starting request: urn:request:27c39edf-1c9d-4c80-8cb1-eef2dad675d4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:37,625 [239] [DEBUG] [app] Starting request: urn:request:63d2e81d-4009-480f-bf26-13622f326ab4 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,628 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:37,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:37,630 [232] [DEBUG] [app] Starting request: urn:request:4eda3d5d-d263-491b-86b5-3487daf91742 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:37,630 [242] [DEBUG] [app] Starting request: urn:request:cfec2720-0368-43e2-b15e-a533c9f5c576 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:37,631 [232] [DEBUG] [app] Ending request: urn:request:4eda3d5d-d263-491b-86b5-3487daf91742 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4eda3d5d-d263-491b-86b5-3487daf91742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:20:37,631 [242] [DEBUG] [app] Ending request: urn:request:cfec2720-0368-43e2-b15e-a533c9f5c576 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cfec2720-0368-43e2-b15e-a533c9f5c576', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:20:37,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:20:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:37,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:20:37,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,633 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:37,635 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:37,635 [244] [DEBUG] [app] Starting request: urn:request:a50a871e-85fd-413a-95bf-6106ef56228a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:37,636 [243] [DEBUG] [app] Starting request: urn:request:ae7baf27-02ec-4caf-af9c-0b0610a7896a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:37,636 [244] [DEBUG] [app] Ending request: urn:request:a50a871e-85fd-413a-95bf-6106ef56228a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a50a871e-85fd-413a-95bf-6106ef56228a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:37,636 [243] [DEBUG] [app] Ending request: urn:request:ae7baf27-02ec-4caf-af9c-0b0610a7896a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae7baf27-02ec-4caf-af9c-0b0610a7896a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:37,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:37,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:37,636 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:37,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:37,641 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:37,648 [241] [DEBUG] [app] Ending request: urn:request:27c39edf-1c9d-4c80-8cb1-eef2dad675d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27c39edf-1c9d-4c80-8cb1-eef2dad675d4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:37,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:37,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:20:37,650 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:37,656 [239] [DEBUG] [app] Ending request: urn:request:63d2e81d-4009-480f-bf26-13622f326ab4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63d2e81d-4009-480f-bf26-13622f326ab4', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:37,656 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:37,656 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) globalpromstats stdout | 2023-01-17 06:20:37,747 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:20:37,750 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 06:20:38,570 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:20:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:43.935629+00:00 (in 5.200119 seconds) notificationworker stdout | 2023-01-17 06:20:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:48 UTC)" (scheduled at 2023-01-17 06:20:38.735066+00:00) notificationworker stdout | 2023-01-17 06:20:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 38, 735892), True, datetime.datetime(2023, 1, 17, 6, 20, 38, 735892), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:20:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:20:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:48.738458+00:00 (in 9.707234 seconds) repositorygcworker stdout | 2023-01-17 06:20:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:39 UTC)" (scheduled at 2023-01-17 06:20:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:20:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:20:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 39, 31691), True, datetime.datetime(2023, 1, 17, 6, 20, 39, 31691), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:20:39,041 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:20:39,041 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:20:39,041 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:20:39,152 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-web stdout | 2023-01-17 06:20:39,154 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:20:39,157 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:20:39,160 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:20:39,163 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:20:39,165 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:20:39,168 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:20:39,170 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:20:39,172 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:20:39,175 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:20:39,178 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:20:39,180 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:20:39,183 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:20:39,185 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:20:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:52.837365+00:00 (in 13.606913 seconds) securityworker stdout | 2023-01-17 06:20:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:09 UTC)" (scheduled at 2023-01-17 06:20:39.230038+00:00) securityworker stdout | 2023-01-17 06:20:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:20:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:20:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:39,243 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:20:39,243 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:09 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:20:39,412 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:20:39,468 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:20:40,518 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:20:41,216 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:20:41,986 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gcworker stdout | 2023-01-17 06:20:41,988 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:20:42,013 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:20:42,078 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:20:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:48.735066+00:00 (in 4.798998 seconds) notificationworker stdout | 2023-01-17 06:20:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:43 UTC)" (scheduled at 2023-01-17 06:20:43.935629+00:00) notificationworker stdout | 2023-01-17 06:20:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:20:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:43 UTC)" executed successfully builder stdout | 2023-01-17 06:20:47,567 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:47,577 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:47,577 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:47,582 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:20:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:20:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:59.337413+00:00 (in 11.404462 seconds) namespacegcworker stdout | 2023-01-17 06:20:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:47 UTC)" (scheduled at 2023-01-17 06:20:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:20:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:20:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:20:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:48,736 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:58.735066+00:00 (in 9.999209 seconds) notificationworker stdout | 2023-01-17 06:20:48,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:58 UTC)" (scheduled at 2023-01-17 06:20:48.735066+00:00) notificationworker stdout | 2023-01-17 06:20:48,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:48,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 48, 736270), True, datetime.datetime(2023, 1, 17, 6, 20, 48, 736270), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:20:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:20:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:39.030814+00:00 (in 50.291882 seconds) repositorygcworker stdout | 2023-01-17 06:20:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:48 UTC)" (scheduled at 2023-01-17 06:20:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:20:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:20:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:20:48,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:48,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:48,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:20:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:20:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:24.641816+00:00 (in 35.811056 seconds) exportactionlogsworker stdout | 2023-01-17 06:20:48,831 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:48 UTC)" (scheduled at 2023-01-17 06:20:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:20:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:20:48,832 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 48, 831197), True, datetime.datetime(2023, 1, 17, 6, 20, 48, 831197), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:20:48,839 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:20:48,839 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:20:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:20:52,625 [244] [DEBUG] [app] Starting request: urn:request:3bf81ec6-c2eb-4027-9ab9-7051144f02f2 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:52,625 [243] [DEBUG] [app] Starting request: urn:request:9288071e-7bf6-480a-a117-97bb4ee9fd4e (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:20:52,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:52,630 [232] [DEBUG] [app] Starting request: urn:request:403faf13-4e84-4867-9784-4f6604dc43ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:52,630 [242] [DEBUG] [app] Starting request: urn:request:fa457305-c79f-484d-b8bf-cf845229124b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:52,630 [232] [DEBUG] [app] Ending request: urn:request:403faf13-4e84-4867-9784-4f6604dc43ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:403faf13-4e84-4867-9784-4f6604dc43ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:20:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:20:52,631 [242] [DEBUG] [app] Ending request: urn:request:fa457305-c79f-484d-b8bf-cf845229124b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fa457305-c79f-484d-b8bf-cf845229124b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:20:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:20:52,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:20:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:52,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:52,636 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:52,636 [243] [DEBUG] [app] Starting request: urn:request:4cab5339-b956-4b77-aa73-0b557c22573b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:52,637 [239] [DEBUG] [app] Starting request: urn:request:98eebba7-1e6d-4b1a-8ac1-fcd61989ce63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:52,637 [243] [DEBUG] [app] Ending request: urn:request:4cab5339-b956-4b77-aa73-0b557c22573b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4cab5339-b956-4b77-aa73-0b557c22573b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:52,637 [239] [DEBUG] [app] Ending request: urn:request:98eebba7-1e6d-4b1a-8ac1-fcd61989ce63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98eebba7-1e6d-4b1a-8ac1-fcd61989ce63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:52,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:52,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:52,643 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:52,643 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:52,648 [243] [DEBUG] [app] Ending request: urn:request:9288071e-7bf6-480a-a117-97bb4ee9fd4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9288071e-7bf6-480a-a117-97bb4ee9fd4e', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:52,648 [244] [DEBUG] [app] Ending request: urn:request:3bf81ec6-c2eb-4027-9ab9-7051144f02f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3bf81ec6-c2eb-4027-9ab9-7051144f02f2', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:52,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.024) gunicorn-web stdout | 2023-01-17 06:20:52,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:52,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:20:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:09.230038+00:00 (in 16.392295 seconds) securityworker stdout | 2023-01-17 06:20:52,837 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:22 UTC)" (scheduled at 2023-01-17 06:20:52.837365+00:00) securityworker stdout | 2023-01-17 06:20:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:52,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:20:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:20:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:27.226424+00:00 (in 29.999557 seconds) buildlogsarchiver stdout | 2023-01-17 06:20:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:27 UTC)" (scheduled at 2023-01-17 06:20:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:20:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 20, 57, 227200), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:20:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:20:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:20:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:27 UTC)" executed successfully builder stdout | 2023-01-17 06:20:57,584 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:57,593 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:57,593 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:57,599 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:08.735066+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 06:20:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:08 UTC)" (scheduled at 2023-01-17 06:20:58.735066+00:00) notificationworker stdout | 2023-01-17 06:20:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 58, 735968), True, datetime.datetime(2023, 1, 17, 6, 20, 58, 735968), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:20:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:20:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:47.932490+00:00 (in 48.594579 seconds) namespacegcworker stdout | 2023-01-17 06:20:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:59 UTC)" (scheduled at 2023-01-17 06:20:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:20:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:20:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 59, 338363), True, datetime.datetime(2023, 1, 17, 6, 20, 59, 338363), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:20:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:20:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:20:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:21:02,934 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:21:03,255 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:21:03,466 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:21:04,325 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:21:04,385 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:21:05,538 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:21:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:21:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:35.848675+00:00 (in 29.999537 seconds) gcworker stdout | 2023-01-17 06:21:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:35 UTC)" (scheduled at 2023-01-17 06:21:05.848675+00:00) gcworker stdout | 2023-01-17 06:21:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726865849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:21:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:21:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:35 UTC)" executed successfully builder stdout | 2023-01-17 06:21:06,294 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:21:06,440 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:21:06,635 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:21:06,638 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:21:06,640 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:21:06,708 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:21:07,599 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:07,608 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:07,608 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:07,614 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:07,625 [239] [DEBUG] [app] Starting request: urn:request:e60a057d-f49b-4171-b2fe-ac592ef4e51c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:07,625 [243] [DEBUG] [app] Starting request: urn:request:8c0790bc-9d81-429b-a162-62ff1a32a6f1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:07,626 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:07,630 [232] [DEBUG] [app] Starting request: urn:request:5fef2950-8e8a-4255-a8e0-2934e63bf687 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:07,630 [236] [DEBUG] [app] Starting request: urn:request:8e6744e3-7c94-45db-96dd-7aea5dfd4880 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:07,630 [232] [DEBUG] [app] Ending request: urn:request:5fef2950-8e8a-4255-a8e0-2934e63bf687 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5fef2950-8e8a-4255-a8e0-2934e63bf687', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:21:07,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:21:07,631 [236] [DEBUG] [app] Ending request: urn:request:8e6744e3-7c94-45db-96dd-7aea5dfd4880 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8e6744e3-7c94-45db-96dd-7aea5dfd4880', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:07,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:21:07,631 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:07,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:07,635 [241] [DEBUG] [app] Starting request: urn:request:02343da4-727d-46b3-9ba4-9da9e50a1cab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:07,635 [243] [DEBUG] [app] Starting request: urn:request:9dabf32a-1fbe-4168-a800-cfc930da3964 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:07,635 [241] [DEBUG] [app] Ending request: urn:request:02343da4-727d-46b3-9ba4-9da9e50a1cab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02343da4-727d-46b3-9ba4-9da9e50a1cab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:07,635 [243] [DEBUG] [app] Ending request: urn:request:9dabf32a-1fbe-4168-a800-cfc930da3964 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9dabf32a-1fbe-4168-a800-cfc930da3964', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:07,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:07,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:07,636 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:07,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:07,636 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:07,640 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:07,646 [243] [DEBUG] [app] Ending request: urn:request:8c0790bc-9d81-429b-a162-62ff1a32a6f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8c0790bc-9d81-429b-a162-62ff1a32a6f1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:07,646 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:07,646 [239] [DEBUG] [app] Ending request: urn:request:e60a057d-f49b-4171-b2fe-ac592ef4e51c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e60a057d-f49b-4171-b2fe-ac592ef4e51c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:07,646 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:21:07,646 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 06:21:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) globalpromstats stdout | 2023-01-17 06:21:07,775 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} repositoryactioncounter stdout | 2023-01-17 06:21:07,785 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} chunkcleanupworker stdout | 2023-01-17 06:21:08,606 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:21:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:18.735066+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 06:21:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:18 UTC)" (scheduled at 2023-01-17 06:21:08.735066+00:00) notificationworker stdout | 2023-01-17 06:21:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 8, 735892), True, datetime.datetime(2023, 1, 17, 6, 21, 8, 735892), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:09,161 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:21:09,165 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:21:09,167 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:21:09,169 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:21:09,172 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:21:09,174 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:21:09,177 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:21:09,179 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:21:09,182 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:21:09,184 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:21:09,186 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:21:09,189 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:21:09,191 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} gunicorn-registry stdout | 2023-01-17 06:21:09,194 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:21:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:22.837365+00:00 (in 13.606915 seconds) securityworker stdout | 2023-01-17 06:21:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:39 UTC)" (scheduled at 2023-01-17 06:21:09.230038+00:00) securityworker stdout | 2023-01-17 06:21:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:21:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:21:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:21:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:21:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:21:09,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:09,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:21:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:21:09,428 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:21:09,503 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:21:10,553 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:21:11,242 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:21:12,021 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gcworker stdout | 2023-01-17 06:21:12,024 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:21:12,029 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:21:12,113 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:21:17,624 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:17,632 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:17,633 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:17,638 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:28.735066+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 06:21:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:28 UTC)" (scheduled at 2023-01-17 06:21:18.735066+00:00) notificationworker stdout | 2023-01-17 06:21:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 18, 735923), True, datetime.datetime(2023, 1, 17, 6, 21, 18, 735923), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:18,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:18,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:18,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:21:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:21:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:25.334138+00:00 (in 5.091497 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:21:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:20 UTC)" (scheduled at 2023-01-17 06:21:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:21:20,242 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:21:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 20, 243009), True, datetime.datetime(2023, 1, 17, 6, 21, 20, 243009), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:21:20,250 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:21:20,250 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:21:20,250 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:22,626 [239] [DEBUG] [app] Starting request: urn:request:36919a47-05ce-4e4e-8167-f4085dd3c811 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:22,626 [243] [DEBUG] [app] Starting request: urn:request:253dd269-e120-41d8-b0c1-b6c0720a1375 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:22,631 [232] [DEBUG] [app] Starting request: urn:request:80060b86-0bf3-4094-b367-ab541c8c2d68 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:22,631 [242] [DEBUG] [app] Starting request: urn:request:de114ffc-aa48-48f8-a2b3-f1523660853c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:22,631 [232] [DEBUG] [app] Ending request: urn:request:80060b86-0bf3-4094-b367-ab541c8c2d68 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:80060b86-0bf3-4094-b367-ab541c8c2d68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:21:22,632 [242] [DEBUG] [app] Ending request: urn:request:de114ffc-aa48-48f8-a2b3-f1523660853c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de114ffc-aa48-48f8-a2b3-f1523660853c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:22,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:21:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:21:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:22,636 [241] [DEBUG] [app] Starting request: urn:request:00e43890-e55c-417c-810a-58679e8ea887 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:22,636 [243] [DEBUG] [app] Starting request: urn:request:a008898a-8672-4bea-a89a-54530528e1b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:22,636 [243] [DEBUG] [app] Ending request: urn:request:a008898a-8672-4bea-a89a-54530528e1b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a008898a-8672-4bea-a89a-54530528e1b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:22,636 [241] [DEBUG] [app] Ending request: urn:request:00e43890-e55c-417c-810a-58679e8ea887 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00e43890-e55c-417c-810a-58679e8ea887', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:22,636 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:22,636 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:22,647 [243] [DEBUG] [app] Ending request: urn:request:253dd269-e120-41d8-b0c1-b6c0720a1375 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:253dd269-e120-41d8-b0c1-b6c0720a1375', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:22,648 [239] [DEBUG] [app] Ending request: urn:request:36919a47-05ce-4e4e-8167-f4085dd3c811 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:36919a47-05ce-4e4e-8167-f4085dd3c811', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:21:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) securityworker stdout | 2023-01-17 06:21:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:39.230038+00:00 (in 16.392249 seconds) securityworker stdout | 2023-01-17 06:21:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:52 UTC)" (scheduled at 2023-01-17 06:21:22.837365+00:00) securityworker stdout | 2023-01-17 06:21:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:21:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:21:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:44.641497+00:00 (in 19.999169 seconds) exportactionlogsworker stdout | 2023-01-17 06:21:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:24 UTC)" (scheduled at 2023-01-17 06:21:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:21:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:21:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:21:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:21:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:39.333860+00:00 (in 13.999289 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:21:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:25 UTC)" (scheduled at 2023-01-17 06:21:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:21:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:21:25,334 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:21:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:21:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:57.226424+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 06:21:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:57 UTC)" (scheduled at 2023-01-17 06:21:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:21:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 21, 27, 227244), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:21:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:21:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:21:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:57 UTC)" executed successfully builder stdout | 2023-01-17 06:21:27,648 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:27,658 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:27,658 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:27,664 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:38.735066+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 06:21:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:38 UTC)" (scheduled at 2023-01-17 06:21:28.735066+00:00) notificationworker stdout | 2023-01-17 06:21:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 28, 735929), True, datetime.datetime(2023, 1, 17, 6, 21, 28, 735929), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:28,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:28,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:21:32,970 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:21:33,291 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:21:33,488 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:21:34,339 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:21:34,420 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:21:35,574 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:21:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:21:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:05.848675+00:00 (in 29.999552 seconds) gcworker stdout | 2023-01-17 06:21:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:05 UTC)" (scheduled at 2023-01-17 06:21:35.848675+00:00) gcworker stdout | 2023-01-17 06:21:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726895849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:21:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:21:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:05 UTC)" executed successfully builder stdout | 2023-01-17 06:21:36,330 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:21:36,446 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:21:36,642 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:21:36,645 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:21:36,647 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:21:36,741 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 06:21:37,625 [239] [DEBUG] [app] Starting request: urn:request:9ffd1a01-4441-41a0-afd4-b454f1829531 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:37,625 [244] [DEBUG] [app] Starting request: urn:request:e57a3884-e53a-41a1-8550-5ccc9cbe2218 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,626 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,628 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:37,629 [240] [DEBUG] [app] Starting request: urn:request:7ca5c712-c9b3-4894-8a7f-cacfb56cb9c9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,630 [240] [DEBUG] [app] Ending request: urn:request:7ca5c712-c9b3-4894-8a7f-cacfb56cb9c9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ca5c712-c9b3-4894-8a7f-cacfb56cb9c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,630 [232] [DEBUG] [app] Starting request: urn:request:7d502d88-4eaa-49fd-843f-a881b02157b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,630 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:21:37,630 [232] [DEBUG] [app] Ending request: urn:request:7d502d88-4eaa-49fd-843f-a881b02157b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d502d88-4eaa-49fd-843f-a881b02157b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:37,630 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:21:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:37,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:37,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:37,635 [244] [DEBUG] [app] Starting request: urn:request:22fcdc9b-c752-4846-a6e7-a80518e4ab1d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:37,636 [244] [DEBUG] [app] Ending request: urn:request:22fcdc9b-c752-4846-a6e7-a80518e4ab1d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:22fcdc9b-c752-4846-a6e7-a80518e4ab1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:37,636 [243] [DEBUG] [app] Starting request: urn:request:d0dc811e-21ae-4301-bb2f-91e94f754e9b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:37,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:21:37,636 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:37,636 [243] [DEBUG] [app] Ending request: urn:request:d0dc811e-21ae-4301-bb2f-91e94f754e9b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d0dc811e-21ae-4301-bb2f-91e94f754e9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:37,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:37,647 [244] [DEBUG] [app] Ending request: urn:request:e57a3884-e53a-41a1-8550-5ccc9cbe2218 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e57a3884-e53a-41a1-8550-5ccc9cbe2218', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:37,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:21:37,647 [239] [DEBUG] [app] Ending request: urn:request:9ffd1a01-4441-41a0-afd4-b454f1829531 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ffd1a01-4441-41a0-afd4-b454f1829531', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:21:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.025) builder stdout | 2023-01-17 06:21:37,664 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:37,673 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:37,673 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:37,678 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:21:37,797 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:21:37,806 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:21:38,623 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:21:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:43.935629+00:00 (in 5.200155 seconds) notificationworker stdout | 2023-01-17 06:21:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:48 UTC)" (scheduled at 2023-01-17 06:21:38.735066+00:00) notificationworker stdout | 2023-01-17 06:21:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 38, 735877), True, datetime.datetime(2023, 1, 17, 6, 21, 38, 735877), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:38,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:38,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:21:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:21:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:48.738458+00:00 (in 9.707233 seconds) repositorygcworker stdout | 2023-01-17 06:21:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:39 UTC)" (scheduled at 2023-01-17 06:21:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:21:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:21:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 39, 31665), True, datetime.datetime(2023, 1, 17, 6, 21, 39, 31665), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:21:39,040 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:21:39,040 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:21:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:39 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:39,169 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:21:39,173 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:21:39,175 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:21:39,178 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:21:39,180 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:21:39,183 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:21:39,186 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-web stdout | 2023-01-17 06:21:39,189 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:21:39,191 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:21:39,193 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:21:39,196 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:21:39,198 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:21:39,201 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:21:39,204 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} securityworker stdout | 2023-01-17 06:21:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:52.837365+00:00 (in 13.606868 seconds) securityworker stdout | 2023-01-17 06:21:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:09 UTC)" (scheduled at 2023-01-17 06:21:39.230038+00:00) securityworker stdout | 2023-01-17 06:21:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:21:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:21:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:21:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:21:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:21:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:39,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:21:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:09 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:21:39,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:21:39,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:20.242198+00:00 (in 40.907874 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:21:39,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:39 UTC)" (scheduled at 2023-01-17 06:21:39.333860+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:21:39,335 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 21, 39, 334699), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:21:39,341 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 39, 334699), True, datetime.datetime(2023, 1, 17, 6, 21, 39, 334699), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:21:39,344 [86] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 39, 334699), True, datetime.datetime(2023, 1, 17, 6, 21, 39, 334699), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 21, 39, 334699), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:21:39,345 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:21:39,345 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:21:39,464 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:21:39,508 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:21:40,580 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:21:41,252 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:21:42,047 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gcworker stdout | 2023-01-17 06:21:42,054 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:21:42,064 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:21:42,124 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:21:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:48.735066+00:00 (in 4.799001 seconds) notificationworker stdout | 2023-01-17 06:21:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:43 UTC)" (scheduled at 2023-01-17 06:21:43.935629+00:00) notificationworker stdout | 2023-01-17 06:21:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:21:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:43 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:21:44,641 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:21:44,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:48.830274+00:00 (in 4.188341 seconds) exportactionlogsworker stdout | 2023-01-17 06:21:44,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:44 UTC)" (scheduled at 2023-01-17 06:21:44.641497+00:00) exportactionlogsworker stdout | 2023-01-17 06:21:44,642 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 21, 44, 642272), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:21:44,650 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 44, 642272), True, datetime.datetime(2023, 1, 17, 6, 21, 44, 642272), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:21:44,652 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 44, 642272), True, datetime.datetime(2023, 1, 17, 6, 21, 44, 642272), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 21, 44, 642272), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:21:44,653 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:21:44,654 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:44 UTC)" executed successfully builder stdout | 2023-01-17 06:21:47,688 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:47,697 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:47,697 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:47,702 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:21:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:21:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:59.337413+00:00 (in 11.404342 seconds) namespacegcworker stdout | 2023-01-17 06:21:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:47 UTC)" (scheduled at 2023-01-17 06:21:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:21:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:21:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:58.735066+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 06:21:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:58 UTC)" (scheduled at 2023-01-17 06:21:48.735066+00:00) notificationworker stdout | 2023-01-17 06:21:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 48, 735866), True, datetime.datetime(2023, 1, 17, 6, 21, 48, 735866), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:21:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:21:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:04.738062+00:00 (in 15.999156 seconds) repositorygcworker stdout | 2023-01-17 06:21:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:48 UTC)" (scheduled at 2023-01-17 06:21:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:21:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:21:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:21:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:21:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:24.641816+00:00 (in 35.811104 seconds) exportactionlogsworker stdout | 2023-01-17 06:21:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:48 UTC)" (scheduled at 2023-01-17 06:21:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:21:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:21:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 48, 831050), True, datetime.datetime(2023, 1, 17, 6, 21, 48, 831050), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:21:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:21:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:21:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:52,625 [239] [DEBUG] [app] Starting request: urn:request:a5b8ae98-abf3-420a-9a54-274d7b5fa2a9 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:52,625 [243] [DEBUG] [app] Starting request: urn:request:71f84b75-c247-44f3-b238-a56284617438 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:21:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:52,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:52,630 [232] [DEBUG] [app] Starting request: urn:request:d501c041-545f-402c-9c32-45b8c90042e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:52,630 [240] [DEBUG] [app] Starting request: urn:request:e4cdc831-0c97-4510-864b-359033169c64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:52,631 [232] [DEBUG] [app] Ending request: urn:request:d501c041-545f-402c-9c32-45b8c90042e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d501c041-545f-402c-9c32-45b8c90042e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:21:52,631 [240] [DEBUG] [app] Ending request: urn:request:e4cdc831-0c97-4510-864b-359033169c64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e4cdc831-0c97-4510-864b-359033169c64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:21:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:52,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,634 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:52,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:52,636 [241] [DEBUG] [app] Starting request: urn:request:608669d8-7606-4228-a4f9-3a777f47fd48 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:52,636 [244] [DEBUG] [app] Starting request: urn:request:55fd7b24-2dbd-40fb-a129-a8c2d72583f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:52,636 [241] [DEBUG] [app] Ending request: urn:request:608669d8-7606-4228-a4f9-3a777f47fd48 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:608669d8-7606-4228-a4f9-3a777f47fd48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:52,637 [244] [DEBUG] [app] Ending request: urn:request:55fd7b24-2dbd-40fb-a129-a8c2d72583f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55fd7b24-2dbd-40fb-a129-a8c2d72583f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:52,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:21:52,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:52,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:52,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:52,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:52,648 [239] [DEBUG] [app] Ending request: urn:request:a5b8ae98-abf3-420a-9a54-274d7b5fa2a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5b8ae98-abf3-420a-9a54-274d7b5fa2a9', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:52,648 [243] [DEBUG] [app] Ending request: urn:request:71f84b75-c247-44f3-b238-a56284617438 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71f84b75-c247-44f3-b238-a56284617438', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:52,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:52,648 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:21:52,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:21:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:21:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:09.230038+00:00 (in 16.392218 seconds) securityworker stdout | 2023-01-17 06:21:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:22 UTC)" (scheduled at 2023-01-17 06:21:52.837365+00:00) securityworker stdout | 2023-01-17 06:21:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:21:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:21:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:27.226424+00:00 (in 29.999548 seconds) buildlogsarchiver stdout | 2023-01-17 06:21:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:27 UTC)" (scheduled at 2023-01-17 06:21:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:21:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 21, 57, 227226), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:21:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:21:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:21:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:27 UTC)" executed successfully builder stdout | 2023-01-17 06:21:57,707 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:57,716 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:57,716 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:57,721 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:08.735066+00:00 (in 9.999552 seconds) notificationworker stdout | 2023-01-17 06:21:58,736 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:08 UTC)" (scheduled at 2023-01-17 06:21:58.735066+00:00) notificationworker stdout | 2023-01-17 06:21:58,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:58,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 58, 736186), True, datetime.datetime(2023, 1, 17, 6, 21, 58, 736186), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:21:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:21:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:47.932490+00:00 (in 48.594648 seconds) namespacegcworker stdout | 2023-01-17 06:21:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:59 UTC)" (scheduled at 2023-01-17 06:21:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:21:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:21:59,338 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 59, 338199), True, datetime.datetime(2023, 1, 17, 6, 21, 59, 338199), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:21:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:21:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:21:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:22:02,981 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:22:03,326 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:22:03,523 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:22:04,349 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:22:04,454 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:22:04,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:22:04,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:39.030814+00:00 (in 34.292331 seconds) repositorygcworker stdout | 2023-01-17 06:22:04,738 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:27:04 UTC)" (scheduled at 2023-01-17 06:22:04.738062+00:00) repositorygcworker stdout | 2023-01-17 06:22:04,739 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 22, 4, 738852), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:22:04,746 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 22, 4, 738852), True, datetime.datetime(2023, 1, 17, 6, 22, 4, 738852), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:22:04,748 [85] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 22, 4, 738852), True, datetime.datetime(2023, 1, 17, 6, 22, 4, 738852), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 22, 4, 738852), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:22:04,750 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:22:04,750 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:27:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:22:05,610 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:22:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:22:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:35.848675+00:00 (in 29.999494 seconds) gcworker stdout | 2023-01-17 06:22:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:35 UTC)" (scheduled at 2023-01-17 06:22:05.848675+00:00) gcworker stdout | 2023-01-17 06:22:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726925849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:22:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:22:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:35 UTC)" executed successfully builder stdout | 2023-01-17 06:22:06,341 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:22:06,451 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:22:06,649 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:22:06,651 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:22:06,653 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:22:06,778 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 06:22:07,625 [244] [DEBUG] [app] Starting request: urn:request:773fba98-fb0c-4c7c-b95b-f3936fe2bad5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:07,625 [239] [DEBUG] [app] Starting request: urn:request:54089f0c-4d1c-442c-afc4-60736021b6eb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:07,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:07,630 [232] [DEBUG] [app] Starting request: urn:request:7a9f08e6-a672-4ed1-b30c-3049fc16551c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:07,630 [232] [DEBUG] [app] Ending request: urn:request:7a9f08e6-a672-4ed1-b30c-3049fc16551c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7a9f08e6-a672-4ed1-b30c-3049fc16551c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:07,630 [242] [DEBUG] [app] Starting request: urn:request:7b8748da-abd2-458a-9700-e85223d4afdb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:07,630 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:22:07,631 [242] [DEBUG] [app] Ending request: urn:request:7b8748da-abd2-458a-9700-e85223d4afdb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b8748da-abd2-458a-9700-e85223d4afdb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:07,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:22:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:07,632 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,634 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:07,635 [243] [DEBUG] [app] Starting request: urn:request:158360dd-de6b-41c9-b29f-6c9d7b32936a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:07,635 [243] [DEBUG] [app] Ending request: urn:request:158360dd-de6b-41c9-b29f-6c9d7b32936a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:158360dd-de6b-41c9-b29f-6c9d7b32936a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:22:07,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:07,635 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:07,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:07,636 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:07,636 [243] [DEBUG] [app] Starting request: urn:request:3bdc4cbd-2fd1-45d4-a2fc-b84c0b6c8009 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:07,636 [243] [DEBUG] [app] Ending request: urn:request:3bdc4cbd-2fd1-45d4-a2fc-b84c0b6c8009 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3bdc4cbd-2fd1-45d4-a2fc-b84c0b6c8009', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:07,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:07,641 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:07,646 [244] [DEBUG] [app] Ending request: urn:request:773fba98-fb0c-4c7c-b95b-f3936fe2bad5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:773fba98-fb0c-4c7c-b95b-f3936fe2bad5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:07,646 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:07,647 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.022) gunicorn-web stdout | 2023-01-17 06:22:07,647 [239] [DEBUG] [app] Ending request: urn:request:54089f0c-4d1c-442c-afc4-60736021b6eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54089f0c-4d1c-442c-afc4-60736021b6eb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) builder stdout | 2023-01-17 06:22:07,723 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:07,730 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:07,730 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:07,736 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:22:07,826 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:22:07,841 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:22:08,656 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:22:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:18.735066+00:00 (in 9.999581 seconds) notificationworker stdout | 2023-01-17 06:22:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:18 UTC)" (scheduled at 2023-01-17 06:22:08.735066+00:00) notificationworker stdout | 2023-01-17 06:22:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 8, 735847), True, datetime.datetime(2023, 1, 17, 6, 22, 8, 735847), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:09,177 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:22:09,180 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:22:09,183 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:22:09,186 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:22:09,189 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:22:09,191 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:22:09,196 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:22:09,199 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:22:09,201 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:22:09,204 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:22:09,206 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:22:09,209 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:22:09,211 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:22:09,214 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} securityworker stdout | 2023-01-17 06:22:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:22.837365+00:00 (in 13.606623 seconds) securityworker stdout | 2023-01-17 06:22:09,231 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:39 UTC)" (scheduled at 2023-01-17 06:22:09.230038+00:00) securityworker stdout | 2023-01-17 06:22:09,231 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:09,231 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:22:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:22:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:22:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:22:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:22:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:22:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:22:09,500 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:22:09,539 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:22:10,616 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:22:11,287 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:22:12,079 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:22:12,082 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gcworker stdout | 2023-01-17 06:22:12,084 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 06:22:12,140 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:22:17,736 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:17,745 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:17,746 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:17,751 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:28.735066+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 06:22:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:28 UTC)" (scheduled at 2023-01-17 06:22:18.735066+00:00) notificationworker stdout | 2023-01-17 06:22:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 18, 735944), True, datetime.datetime(2023, 1, 17, 6, 22, 18, 735944), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:22:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:22:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:25.334138+00:00 (in 5.091432 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:22:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:20 UTC)" (scheduled at 2023-01-17 06:22:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:22:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:22:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 20, 243129), True, datetime.datetime(2023, 1, 17, 6, 22, 20, 243129), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:22:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:22:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:22:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:22,626 [243] [DEBUG] [app] Starting request: urn:request:e2ba072e-3a55-48aa-b657-ade893ca8947 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:22,626 [239] [DEBUG] [app] Starting request: urn:request:d39d78b2-11e0-4b7a-996c-dbd2096339ea (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,628 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:22,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:22,630 [242] [DEBUG] [app] Starting request: urn:request:b8235aeb-7db7-4f2e-b173-b8350d5a6b44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,631 [242] [DEBUG] [app] Ending request: urn:request:b8235aeb-7db7-4f2e-b173-b8350d5a6b44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b8235aeb-7db7-4f2e-b173-b8350d5a6b44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,631 [232] [DEBUG] [app] Starting request: urn:request:442ad478-6cc8-4d19-a6fe-be93396ec4d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:22:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:22:22,632 [232] [DEBUG] [app] Ending request: urn:request:442ad478-6cc8-4d19-a6fe-be93396ec4d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:442ad478-6cc8-4d19-a6fe-be93396ec4d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:22:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:22,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,634 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:22,636 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:22,636 [243] [DEBUG] [app] Starting request: urn:request:d437ed07-4f82-41cc-818a-b75c794f7fb3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:22,636 [243] [DEBUG] [app] Ending request: urn:request:d437ed07-4f82-41cc-818a-b75c794f7fb3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d437ed07-4f82-41cc-818a-b75c794f7fb3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:22,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:22,637 [244] [DEBUG] [app] Starting request: urn:request:a730db0b-bba1-4991-8c1a-821090bf772e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:22,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:22,637 [244] [DEBUG] [app] Ending request: urn:request:a730db0b-bba1-4991-8c1a-821090bf772e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a730db0b-bba1-4991-8c1a-821090bf772e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:22,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:22,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:22,638 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:22,638 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:22,648 [243] [DEBUG] [app] Ending request: urn:request:e2ba072e-3a55-48aa-b657-ade893ca8947 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2ba072e-3a55-48aa-b657-ade893ca8947', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:22,648 [239] [DEBUG] [app] Ending request: urn:request:d39d78b2-11e0-4b7a-996c-dbd2096339ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d39d78b2-11e0-4b7a-996c-dbd2096339ea', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:22,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:22,649 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:22:22,649 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) securityworker stdout | 2023-01-17 06:22:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:39.230038+00:00 (in 16.392201 seconds) securityworker stdout | 2023-01-17 06:22:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:52 UTC)" (scheduled at 2023-01-17 06:22:22.837365+00:00) securityworker stdout | 2023-01-17 06:22:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:22,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:22:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:22:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:48.830274+00:00 (in 24.188010 seconds) exportactionlogsworker stdout | 2023-01-17 06:22:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:24 UTC)" (scheduled at 2023-01-17 06:22:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:22:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:22:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:22:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:22:25,335 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:20.242198+00:00 (in 54.907322 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:22:25,335 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:25 UTC)" (scheduled at 2023-01-17 06:22:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:22:25,335 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:22:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:22:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:22:27,226 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:57.226424+00:00 (in 29.999580 seconds) buildlogsarchiver stdout | 2023-01-17 06:22:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:57 UTC)" (scheduled at 2023-01-17 06:22:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:22:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 22, 27, 227189), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:22:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:22:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:22:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:57 UTC)" executed successfully builder stdout | 2023-01-17 06:22:27,753 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:27,762 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:27,762 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:27,768 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:38.735066+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 06:22:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:38 UTC)" (scheduled at 2023-01-17 06:22:28.735066+00:00) notificationworker stdout | 2023-01-17 06:22:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 28, 735940), True, datetime.datetime(2023, 1, 17, 6, 22, 28, 735940), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:22:33,016 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:22:33,361 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:22:33,558 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:22:34,385 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:22:34,480 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:22:35,644 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:22:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:22:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:05.848675+00:00 (in 29.999535 seconds) gcworker stdout | 2023-01-17 06:22:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:05 UTC)" (scheduled at 2023-01-17 06:22:35.848675+00:00) gcworker stdout | 2023-01-17 06:22:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726955849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:22:35,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:22:35,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:35,857 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:05 UTC)" executed successfully builder stdout | 2023-01-17 06:22:36,349 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:22:36,486 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:22:36,655 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:22:36,657 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:22:36,660 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:22:36,813 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 06:22:37,625 [239] [DEBUG] [app] Starting request: urn:request:e425c1f5-62be-496a-9b9f-79bbc13e54e7 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:37,625 [244] [DEBUG] [app] Starting request: urn:request:a3dd7b28-b344-41cd-9721-89862549a447 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:37,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:37,631 [232] [DEBUG] [app] Starting request: urn:request:714eb740-d766-4e42-b8a1-ae454237b9a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,631 [232] [DEBUG] [app] Ending request: urn:request:714eb740-d766-4e42-b8a1-ae454237b9a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:714eb740-d766-4e42-b8a1-ae454237b9a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,631 [242] [DEBUG] [app] Starting request: urn:request:31def0bb-ff2e-4ad4-b726-2ef5ffc80520 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:22:37,632 [242] [DEBUG] [app] Ending request: urn:request:31def0bb-ff2e-4ad4-b726-2ef5ffc80520 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31def0bb-ff2e-4ad4-b726-2ef5ffc80520', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:22:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:22:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:37,636 [241] [DEBUG] [app] Starting request: urn:request:2cfb0506-3f6a-4a75-86bb-9d8d2ccdd217 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:37,636 [239] [DEBUG] [app] Starting request: urn:request:5632cb1f-69a1-4daa-a029-a7b6bb52cc1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:37,636 [241] [DEBUG] [app] Ending request: urn:request:2cfb0506-3f6a-4a75-86bb-9d8d2ccdd217 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cfb0506-3f6a-4a75-86bb-9d8d2ccdd217', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:37,637 [239] [DEBUG] [app] Ending request: urn:request:5632cb1f-69a1-4daa-a029-a7b6bb52cc1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5632cb1f-69a1-4daa-a029-a7b6bb52cc1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:37,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:37,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:37,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:37,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:37,647 [244] [DEBUG] [app] Ending request: urn:request:a3dd7b28-b344-41cd-9721-89862549a447 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a3dd7b28-b344-41cd-9721-89862549a447', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:37,647 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:37,648 [239] [DEBUG] [app] Ending request: urn:request:e425c1f5-62be-496a-9b9f-79bbc13e54e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e425c1f5-62be-496a-9b9f-79bbc13e54e7', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:37,648 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:22:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 06:22:37,776 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:37,784 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:37,784 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:37,790 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:22:37,843 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:22:37,875 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:22:38,673 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:22:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:43.935629+00:00 (in 5.200132 seconds) notificationworker stdout | 2023-01-17 06:22:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:48 UTC)" (scheduled at 2023-01-17 06:22:38.735066+00:00) notificationworker stdout | 2023-01-17 06:22:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 38, 735952), True, datetime.datetime(2023, 1, 17, 6, 22, 38, 735952), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:38,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:22:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:22:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:48.738458+00:00 (in 9.707192 seconds) repositorygcworker stdout | 2023-01-17 06:22:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:39 UTC)" (scheduled at 2023-01-17 06:22:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:22:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:22:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 39, 31662), True, datetime.datetime(2023, 1, 17, 6, 22, 39, 31662), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:22:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:22:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:22:39,039 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:39 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:39,184 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:22:39,188 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:22:39,191 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:22:39,193 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:22:39,196 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:22:39,198 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:22:39,201 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:22:39,208 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:22:39,211 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:22:39,214 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:22:39,216 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-web stdout | 2023-01-17 06:22:39,219 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:22:39,221 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:22:39,223 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} securityworker stdout | 2023-01-17 06:22:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:52.837365+00:00 (in 13.606892 seconds) securityworker stdout | 2023-01-17 06:22:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:09 UTC)" (scheduled at 2023-01-17 06:22:39.230038+00:00) securityworker stdout | 2023-01-17 06:22:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:22:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:22:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:39,241 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:39,241 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:22:39,241 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:09 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:22:39,525 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:22:39,554 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:22:40,652 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:22:41,323 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:22:42,115 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gcworker stdout | 2023-01-17 06:22:42,118 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} blobuploadcleanupworker stdout | 2023-01-17 06:22:42,120 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 06:22:42,163 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:22:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:48.735066+00:00 (in 4.798966 seconds) notificationworker stdout | 2023-01-17 06:22:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:43 UTC)" (scheduled at 2023-01-17 06:22:43.935629+00:00) notificationworker stdout | 2023-01-17 06:22:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:22:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:43 UTC)" executed successfully builder stdout | 2023-01-17 06:22:47,800 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:47,810 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:47,810 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:47,816 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:22:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:22:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:59.337413+00:00 (in 11.404484 seconds) namespacegcworker stdout | 2023-01-17 06:22:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:47 UTC)" (scheduled at 2023-01-17 06:22:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:22:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:22:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:47 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:22:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:58.735066+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 06:22:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:58 UTC)" (scheduled at 2023-01-17 06:22:48.735066+00:00) notificationworker stdout | 2023-01-17 06:22:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 48, 735877), True, datetime.datetime(2023, 1, 17, 6, 22, 48, 735877), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:22:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:22:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:39.030814+00:00 (in 50.291928 seconds) repositorygcworker stdout | 2023-01-17 06:22:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:48 UTC)" (scheduled at 2023-01-17 06:22:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:22:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:22:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:22:48,747 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:48,747 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:48,747 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:22:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:22:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:24.641816+00:00 (in 35.811071 seconds) exportactionlogsworker stdout | 2023-01-17 06:22:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:48 UTC)" (scheduled at 2023-01-17 06:22:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:22:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:22:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 48, 831092), True, datetime.datetime(2023, 1, 17, 6, 22, 48, 831092), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:22:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:22:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:22:48,838 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:52,625 [241] [DEBUG] [app] Starting request: urn:request:33064b39-6e25-48c0-b5b4-be7fefc0b429 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:52,625 [244] [DEBUG] [app] Starting request: urn:request:5d125a1f-86da-43b2-81d8-091b43c9ea05 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:22:52,626 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,629 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:52,630 [232] [DEBUG] [app] Starting request: urn:request:9986085f-11e6-4832-b809-9aa7b388294f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:52,631 [240] [DEBUG] [app] Starting request: urn:request:b47fa063-2a00-40e2-9fe1-ab0cfaa6cd14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:52,631 [232] [DEBUG] [app] Ending request: urn:request:9986085f-11e6-4832-b809-9aa7b388294f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9986085f-11e6-4832-b809-9aa7b388294f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:22:52,631 [240] [DEBUG] [app] Ending request: urn:request:b47fa063-2a00-40e2-9fe1-ab0cfaa6cd14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b47fa063-2a00-40e2-9fe1-ab0cfaa6cd14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:22:52,631 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:22:52,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:52,631 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:52,632 [241] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,634 [241] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:52,636 [239] [DEBUG] [app] Starting request: urn:request:1392109e-a6c0-45b5-a864-0f8a34cee1e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:52,636 [243] [DEBUG] [app] Starting request: urn:request:644d314b-fb63-4b14-a183-f2f5999ca6e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:52,636 [239] [DEBUG] [app] Ending request: urn:request:1392109e-a6c0-45b5-a864-0f8a34cee1e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1392109e-a6c0-45b5-a864-0f8a34cee1e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:52,636 [243] [DEBUG] [app] Ending request: urn:request:644d314b-fb63-4b14-a183-f2f5999ca6e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:644d314b-fb63-4b14-a183-f2f5999ca6e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:52,637 [241] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:52,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:52,637 [241] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:52,642 [241] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:52,648 [241] [DEBUG] [app] Ending request: urn:request:33064b39-6e25-48c0-b5b4-be7fefc0b429 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33064b39-6e25-48c0-b5b4-be7fefc0b429', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:52,648 [244] [DEBUG] [app] Ending request: urn:request:5d125a1f-86da-43b2-81d8-091b43c9ea05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d125a1f-86da-43b2-81d8-091b43c9ea05', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:52,648 [241] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:52,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:22:52,648 [241] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:22:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:52,838 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:09.230038+00:00 (in 16.392174 seconds) securityworker stdout | 2023-01-17 06:22:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:22 UTC)" (scheduled at 2023-01-17 06:22:52.837365+00:00) securityworker stdout | 2023-01-17 06:22:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:52,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:22 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:22:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:22:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:27.226424+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 06:22:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:27 UTC)" (scheduled at 2023-01-17 06:22:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:22:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 22, 57, 227241), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:22:57,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:22:57,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:22:57,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:27 UTC)" executed successfully builder stdout | 2023-01-17 06:22:57,826 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:57,835 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:57,835 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:57,841 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:08.735066+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 06:22:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:08 UTC)" (scheduled at 2023-01-17 06:22:58.735066+00:00) notificationworker stdout | 2023-01-17 06:22:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 58, 735895), True, datetime.datetime(2023, 1, 17, 6, 22, 58, 735895), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:22:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:22:59,338 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:47.932490+00:00 (in 48.594611 seconds) namespacegcworker stdout | 2023-01-17 06:22:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:59 UTC)" (scheduled at 2023-01-17 06:22:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:22:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:22:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 59, 338296), True, datetime.datetime(2023, 1, 17, 6, 22, 59, 338296), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:22:59,346 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:22:59,346 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:22:59,346 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:23:03,024 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:23:03,381 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:23:03,594 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:23:04,421 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:23:04,499 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:23:05,667 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:23:05,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:23:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:35.848675+00:00 (in 29.999559 seconds) gcworker stdout | 2023-01-17 06:23:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:35 UTC)" (scheduled at 2023-01-17 06:23:05.848675+00:00) gcworker stdout | 2023-01-17 06:23:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726985849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:23:05,857 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:23:05,857 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:35 UTC)" executed successfully builder stdout | 2023-01-17 06:23:06,380 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:23:06,500 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:23:06,662 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:23:06,664 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:23:06,666 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:23:06,848 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 06:23:07,625 [239] [DEBUG] [app] Starting request: urn:request:ed3bd191-30ad-4ebe-aeae-15ccbde32255 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:07,625 [243] [DEBUG] [app] Starting request: urn:request:5471c637-75d2-4754-afe3-2a40eff086b5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:07,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:07,630 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:07,630 [242] [DEBUG] [app] Starting request: urn:request:941e969d-ebc5-4671-9c9f-4dc76572cd57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:07,631 [233] [DEBUG] [app] Starting request: urn:request:4c8ef178-3a9c-45e2-bcbf-3b77d70cc8ed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:07,631 [242] [DEBUG] [app] Ending request: urn:request:941e969d-ebc5-4671-9c9f-4dc76572cd57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:941e969d-ebc5-4671-9c9f-4dc76572cd57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:23:07,631 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:23:07,631 [233] [DEBUG] [app] Ending request: urn:request:4c8ef178-3a9c-45e2-bcbf-3b77d70cc8ed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c8ef178-3a9c-45e2-bcbf-3b77d70cc8ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:07,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:07,631 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:07,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:07,635 [244] [DEBUG] [app] Starting request: urn:request:3dc133b0-e419-4b2b-929d-138166f5b9fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:07,636 [239] [DEBUG] [app] Starting request: urn:request:32bda391-c6b0-46bb-b256-966231098961 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:07,636 [244] [DEBUG] [app] Ending request: urn:request:3dc133b0-e419-4b2b-929d-138166f5b9fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3dc133b0-e419-4b2b-929d-138166f5b9fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:23:07,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:07,636 [239] [DEBUG] [app] Ending request: urn:request:32bda391-c6b0-46bb-b256-966231098961 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32bda391-c6b0-46bb-b256-966231098961', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:07,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:07,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:07,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:07,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:07,641 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:07,641 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:07,647 [239] [DEBUG] [app] Ending request: urn:request:ed3bd191-30ad-4ebe-aeae-15ccbde32255 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed3bd191-30ad-4ebe-aeae-15ccbde32255', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:07,647 [239] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:07,647 [243] [DEBUG] [app] Ending request: urn:request:5471c637-75d2-4754-afe3-2a40eff086b5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5471c637-75d2-4754-afe3-2a40eff086b5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:07,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:07,647 [243] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.023) gunicorn-web stdout | 2023-01-17 06:23:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 06:23:07,851 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:07,860 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:07,860 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:07,865 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:23:07,869 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:23:07,901 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} chunkcleanupworker stdout | 2023-01-17 06:23:08,708 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:23:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:18.735066+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 06:23:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:18 UTC)" (scheduled at 2023-01-17 06:23:08.735066+00:00) notificationworker stdout | 2023-01-17 06:23:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 8, 735902), True, datetime.datetime(2023, 1, 17, 6, 23, 8, 735902), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:08,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:08,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:08,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:09,191 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:23:09,196 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:23:09,199 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-registry stdout | 2023-01-17 06:23:09,204 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-web stdout | 2023-01-17 06:23:09,207 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:23:09,209 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-web stdout | 2023-01-17 06:23:09,211 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:23:09,217 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:23:09,221 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-web stdout | 2023-01-17 06:23:09,224 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:23:09,226 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:23:09,228 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:23:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:22.837365+00:00 (in 13.606889 seconds) securityworker stdout | 2023-01-17 06:23:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:39 UTC)" (scheduled at 2023-01-17 06:23:09.230038+00:00) securityworker stdout | 2023-01-17 06:23:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... gunicorn-registry stdout | 2023-01-17 06:23:09,230 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:23:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:23:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state gunicorn-registry stdout | 2023-01-17 06:23:09,233 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} securityworker stdout | 2023-01-17 06:23:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:09,242 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:23:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:39 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:23:09,560 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:23:09,588 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:23:10,687 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:23:11,345 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:23:12,123 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:23:12,149 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gcworker stdout | 2023-01-17 06:23:12,151 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 06:23:12,199 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} builder stdout | 2023-01-17 06:23:17,875 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:17,885 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:17,885 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:17,891 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:28.735066+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 06:23:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:28 UTC)" (scheduled at 2023-01-17 06:23:18.735066+00:00) notificationworker stdout | 2023-01-17 06:23:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 18, 735918), True, datetime.datetime(2023, 1, 17, 6, 23, 18, 735918), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:23:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:23:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:25.334138+00:00 (in 5.091485 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:23:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:20 UTC)" (scheduled at 2023-01-17 06:23:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:23:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:23:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 20, 243050), True, datetime.datetime(2023, 1, 17, 6, 23, 20, 243050), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:23:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:23:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:23:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:22,625 [239] [DEBUG] [app] Starting request: urn:request:aa22e92d-c750-4ec5-859d-50b0783a575c (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:22,625 [243] [DEBUG] [app] Starting request: urn:request:34e72aa6-0ef0-480c-956d-370df008ae89 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:22,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:22,630 [240] [DEBUG] [app] Starting request: urn:request:3b6d2fda-a544-4bfc-a506-bc1efae5ddd7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,630 [240] [DEBUG] [app] Ending request: urn:request:3b6d2fda-a544-4bfc-a506-bc1efae5ddd7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b6d2fda-a544-4bfc-a506-bc1efae5ddd7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,630 [232] [DEBUG] [app] Starting request: urn:request:c8e9dda1-420c-4388-b982-9c5ebc553c1c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,631 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:22,631 [232] [DEBUG] [app] Ending request: urn:request:c8e9dda1-420c-4388-b982-9c5ebc553c1c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8e9dda1-420c-4388-b982-9c5ebc553c1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:22,631 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:23:22,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:23:22,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:22,632 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,632 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,634 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:22,634 [243] [DEBUG] [app] Starting request: urn:request:5966eaaa-485c-40d9-b850-f036a4cb5183 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:22,635 [243] [DEBUG] [app] Ending request: urn:request:5966eaaa-485c-40d9-b850-f036a4cb5183 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5966eaaa-485c-40d9-b850-f036a4cb5183', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:22,635 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:22,635 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:22,636 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:22,636 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:22,637 [244] [DEBUG] [app] Starting request: urn:request:436ffcd1-8944-4f48-9e4b-da7c3b985574 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:22,637 [244] [DEBUG] [app] Ending request: urn:request:436ffcd1-8944-4f48-9e4b-da7c3b985574 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:436ffcd1-8944-4f48-9e4b-da7c3b985574', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:22,638 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:22,638 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:22,638 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:22,640 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:22,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:22,646 [239] [DEBUG] [app] Ending request: urn:request:aa22e92d-c750-4ec5-859d-50b0783a575c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aa22e92d-c750-4ec5-859d-50b0783a575c', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:22,646 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.022 47 0.022) gunicorn-web stdout | 2023-01-17 06:23:22,647 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:22,648 [243] [DEBUG] [app] Ending request: urn:request:34e72aa6-0ef0-480c-956d-370df008ae89 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:34e72aa6-0ef0-480c-956d-370df008ae89', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:22,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:22,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) securityworker stdout | 2023-01-17 06:23:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:39.230038+00:00 (in 16.392261 seconds) securityworker stdout | 2023-01-17 06:23:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:52 UTC)" (scheduled at 2023-01-17 06:23:22.837365+00:00) securityworker stdout | 2023-01-17 06:23:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:23:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:22,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:22,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:23:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:23:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:48.830274+00:00 (in 24.188051 seconds) exportactionlogsworker stdout | 2023-01-17 06:23:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:24 UTC)" (scheduled at 2023-01-17 06:23:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:23:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:23:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:23:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:23:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:20.242198+00:00 (in 54.907591 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:23:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:25 UTC)" (scheduled at 2023-01-17 06:23:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:23:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:23:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:23:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:23:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:57.226424+00:00 (in 29.999533 seconds) buildlogsarchiver stdout | 2023-01-17 06:23:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:57 UTC)" (scheduled at 2023-01-17 06:23:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:23:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 23, 27, 227220), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:23:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:23:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:23:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:57 UTC)" executed successfully builder stdout | 2023-01-17 06:23:27,901 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:27,909 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:27,909 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:27,915 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:38.735066+00:00 (in 9.999500 seconds) notificationworker stdout | 2023-01-17 06:23:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:38 UTC)" (scheduled at 2023-01-17 06:23:28.735066+00:00) notificationworker stdout | 2023-01-17 06:23:28,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:28,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 28, 735980), True, datetime.datetime(2023, 1, 17, 6, 23, 28, 735980), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:28,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:28,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:28,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:23:33,058 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:23:33,395 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:23:33,629 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:23:34,434 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:23:34,534 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:23:35,702 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:23:35,849 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:23:35,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:05.848675+00:00 (in 29.999478 seconds) gcworker stdout | 2023-01-17 06:23:35,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:05 UTC)" (scheduled at 2023-01-17 06:23:35.848675+00:00) gcworker stdout | 2023-01-17 06:23:35,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672727015849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:23:35,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:23:35,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:35,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:05 UTC)" executed successfully builder stdout | 2023-01-17 06:23:36,408 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:23:36,536 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:23:36,669 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:23:36,671 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:23:36,673 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:23:36,884 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 06:23:37,625 [244] [DEBUG] [app] Starting request: urn:request:a070f947-93a8-4cb8-9968-b3854dec0fbb (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:37,625 [239] [DEBUG] [app] Starting request: urn:request:e53d9887-ace7-4a00-a58f-29ff39fb7a3d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:37,626 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,629 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:37,630 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:37,631 [232] [DEBUG] [app] Starting request: urn:request:134673a9-3fd5-4bf7-971f-61e926ab3b5b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,631 [242] [DEBUG] [app] Starting request: urn:request:8ff3af5b-62fb-4d66-99b2-927cd3ad4afd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,631 [232] [DEBUG] [app] Ending request: urn:request:134673a9-3fd5-4bf7-971f-61e926ab3b5b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:134673a9-3fd5-4bf7-971f-61e926ab3b5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:23:37,632 [242] [DEBUG] [app] Ending request: urn:request:8ff3af5b-62fb-4d66-99b2-927cd3ad4afd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ff3af5b-62fb-4d66-99b2-927cd3ad4afd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:23:37,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:37,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:37,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:37,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:37,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:37,636 [243] [DEBUG] [app] Starting request: urn:request:8d121f14-92cd-4474-9521-949a61c057d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:37,636 [243] [DEBUG] [app] Ending request: urn:request:8d121f14-92cd-4474-9521-949a61c057d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d121f14-92cd-4474-9521-949a61c057d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:37,636 [244] [DEBUG] [app] Starting request: urn:request:1baeb208-642a-4903-9566-e0a0f67eab08 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:37,637 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:37,637 [244] [DEBUG] [app] Ending request: urn:request:1baeb208-642a-4903-9566-e0a0f67eab08 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1baeb208-642a-4903-9566-e0a0f67eab08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:37,637 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:23:37,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:37,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:37,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:37,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:37,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:37,643 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:37,648 [244] [DEBUG] [app] Ending request: urn:request:a070f947-93a8-4cb8-9968-b3854dec0fbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a070f947-93a8-4cb8-9968-b3854dec0fbb', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:37,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:23:37,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:37,648 [239] [DEBUG] [app] Ending request: urn:request:e53d9887-ace7-4a00-a58f-29ff39fb7a3d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e53d9887-ace7-4a00-a58f-29ff39fb7a3d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:37,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:23:37,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" repositoryactioncounter stdout | 2023-01-17 06:23:37,885 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} builder stdout | 2023-01-17 06:23:37,920 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:37,935 [60] [DEBUG] [buildman.manager.ephemeral] Creating build job for build 582cef57-43a5-43d1-9ca7-1d229e4a10d8 globalpromstats stdout | 2023-01-17 06:23:37,936 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:23:37,937 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 1, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": null}"} builder stdout | 2023-01-17 06:23:37,937 [60] [DEBUG] [buildman.manager.ephemeral] Scheduling build job buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8 builder stdout | 2023-01-17 06:23:37,937 [60] [DEBUG] [buildman.manager.ephemeral] Scheduling build 582cef57-43a5-43d1-9ca7-1d229e4a10d8 builder stdout | 2023-01-17 06:23:37,937 [60] [DEBUG] [buildman.manager.executor] Using service account token for Kubernetes authentication builder stdout | 2023-01-17 06:23:37,937 [60] [DEBUG] [buildman.manager.executor] Executor config: {'BUILDER_CONTAINER_IMAGE': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.1', 'BUILDER_NAMESPACE': 'virtual-builds', 'CONTAINER_CPU_LIMITS': '1000m', 'CONTAINER_CPU_REQUEST': '500m', 'CONTAINER_MEMORY_LIMITS': '1G', 'CONTAINER_MEMORY_REQUEST': '1G', 'DEBUG': True, 'EXECUTOR': 'kubernetesPodman', 'K8S_API_SERVER': 'api.whu411az22.qe.azure.devcluster.openshift.com:6443', 'K8S_API_TLS_CA': '/conf/stack/extra_ca_certs/build_cluster.crt', 'KUBERNETES_DISTRIBUTION': 'openshift', 'NAME': 'openshift', 'NODE_SELECTOR_LABEL_KEY': '', 'NODE_SELECTOR_LABEL_VALUE': '', 'QUAY_PASSWORD': 'eyJhbGciOiJSUzUxMiJ9.eyJzdWIiOiI3YTE0OWIwNWZkZTI0ZWMwYWQ3ZTIwMjQwZmMyYmU2MCJ9.W-4araN4Cus25Nl39ZOp5MmTZBzeJQlGVhaxh1bRWX2hcAhCaoflBf4l2U71ewD16F9My20HPSSpqIJJ1zydnrXKA731WRknwHYetVZ929_vVNUtHcbCvDYTzCseg781DihDbiWm2pK6mbUKLUEXJXzVaAo7hTbXQsgs63EpKGF3bJyN0M76ZnU8BNkMkaj_cWN0ir8As1vf_OEJudIiSmxxUruF-VMANbTkE50UlP-oh2VCqypBxQ81R5VLu4YsdXfNUwl5jdl14T9MGK0OTw7WAt5VdHBQqr3lsjd4WDWJqigemyrdYM5BC1XZXh8j8v_CfU-yaKXNugTtr8LYk1n6Wso1ThxL9eDUfYFvLRYLoJYizGfOjPcoPJugRIh-tS6ki7zy33-Ya4pkSLLX-OdN4Vq1UJvfE13X8PAOSAbWKF7-7dk1DG2LNW1M2UtEelmyVg6KQQ7cRlmMXp7QouaKO9vDKmudDH_2H9ZVrsXhSFTNmoS_YVkDbxqTs6ugtx5TLxbKYYdNnUygDVKprNdRZSZc-adH4JPuRBroA6CTup13PyW92EtePRN6G2IpXInMC9kkApur5qUynddIscb5KnCaYuKNeqQBCQDI7YxP6CCAfm1obf1nUWbDc2uk4DSNitRZJ84hsLBCLTABTyR0aCGYszCeU4zQ-rn5cc8', 'QUAY_USERNAME': '|76db245a-0d75-11e7-a544-28d244ea5a6d.j9is.751956', 'SERVICE_ACCOUNT_NAME': 'quay-builder', 'SERVICE_ACCOUNT_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6ImJFWkNENWJMMFZKVXdYcW8wVUJmUDktNWxiMGI2MHhBcG5vNElQZXZ3WWMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY3NDAxNzMxOCwiaWF0IjoxNjczOTMwOTE4LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjJhYWZmMGYwLWQ0YjctNDBkMC05NTYxLWU2OWVmNWFjNDAzMCJ9fSwibmJmIjoxNjczOTMwOTE4LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.rxgztiMRRX17HbDGwXvCOAFLYwuIFIoj_0IHKn9Rjvku5WcW76NZFLJ8LkTAeg4mW56vxDmsE-Vvz2a60PhsZbiAQRGmHcYYo36ErCJ-OzTJ7g7qkUx-arCxZjX-CjDI7DlgkC_8BepdpfonxaKiz9NQLH_-6qTu9GOIzxPLueHUfGDIPYqFJQUJkz6c1fxiyUR2J2L0KGEmsPPX2WzwrG0uyUvnup_6nShOoAfXUYAB8L3EXNrKEQaa26FWkuJzDwwg-x50socav-KI68gHAd_ptlyoyUWTFmrRKPHeruKntFC0skPZ7swnwToRKsqGPNEr9dcIxy9kVoEXJUMGV1c9kMN6zFFrBGkBZHo0jv4Ju6sHqYYj8TwnYnax_y_SobBxJ01MGcDIEJX3A60EYiy1_aC5gBm3_uGtpPWGbBqWLzacUcyhKWXfajWNFN_3HHyRu6M5nuQbA3Y_m3R283kaCcsuMMPfGzLQc8lO7vlyyIh4M6VbHpXYVbGExU488jiDC9AoMrXmkLVCmlim0V9oNOx_ZrKIzF53iHVogBATa4n3gYVuDNyK1ywUiM7KNp375vTMYdKdtS4Qen_T838AZuutlVdNeDNSdEHfzEYVLJ42Qky3uqbiAh_7aXaXNK0-Yjp3Df_myEj4M25s7MaUdmU3vfUwEnn40fmM1Ks', 'SETUP_TIME': 180, 'VOLUME_SIZE': '8G'} builder stdout | 2023-01-17 06:23:37,937 [60] [DEBUG] [buildman.manager.executor] Kubernetes request: GET https://api.whu411az22.qe.azure.devcluster.openshift.com:6443/apis/batch/v1/namespaces/virtual-builds/jobs: {'params': {'labelSelector': 'build,time,manager,quay-sha'}, 'verify': '/conf/stack/extra_ca_certs/build_cluster.crt', 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJFWkNENWJMMFZKVXdYcW8wVUJmUDktNWxiMGI2MHhBcG5vNElQZXZ3WWMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY3NDAxNzMxOCwiaWF0IjoxNjczOTMwOTE4LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjJhYWZmMGYwLWQ0YjctNDBkMC05NTYxLWU2OWVmNWFjNDAzMCJ9fSwibmJmIjoxNjczOTMwOTE4LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.rxgztiMRRX17HbDGwXvCOAFLYwuIFIoj_0IHKn9Rjvku5WcW76NZFLJ8LkTAeg4mW56vxDmsE-Vvz2a60PhsZbiAQRGmHcYYo36ErCJ-OzTJ7g7qkUx-arCxZjX-CjDI7DlgkC_8BepdpfonxaKiz9NQLH_-6qTu9GOIzxPLueHUfGDIPYqFJQUJkz6c1fxiyUR2J2L0KGEmsPPX2WzwrG0uyUvnup_6nShOoAfXUYAB8L3EXNrKEQaa26FWkuJzDwwg-x50socav-KI68gHAd_ptlyoyUWTFmrRKPHeruKntFC0skPZ7swnwToRKsqGPNEr9dcIxy9kVoEXJUMGV1c9kMN6zFFrBGkBZHo0jv4Ju6sHqYYj8TwnYnax_y_SobBxJ01MGcDIEJX3A60EYiy1_aC5gBm3_uGtpPWGbBqWLzacUcyhKWXfajWNFN_3HHyRu6M5nuQbA3Y_m3R283kaCcsuMMPfGzLQc8lO7vlyyIh4M6VbHpXYVbGExU488jiDC9AoMrXmkLVCmlim0V9oNOx_ZrKIzF53iHVogBATa4n3gYVuDNyK1ywUiM7KNp375vTMYdKdtS4Qen_T838AZuutlVdNeDNSdEHfzEYVLJ42Qky3uqbiAh_7aXaXNK0-Yjp3Df_myEj4M25s7MaUdmU3vfUwEnn40fmM1Ks'}} builder stdout | 2023-01-17 06:23:37,939 [60] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.whu411az22.qe.azure.devcluster.openshift.com:6443 builder stdout | 2023-01-17 06:23:37,955 [60] [DEBUG] [urllib3.connectionpool] https://api.whu411az22.qe.azure.devcluster.openshift.com:6443 "GET /apis/batch/v1/namespaces/virtual-builds/jobs?labelSelector=build%2Ctime%2Cmanager%2Cquay-sha HTTP/1.1" 200 94 builder stdout | 2023-01-17 06:23:37,955 [60] [DEBUG] [buildman.manager.executor] Kubernetes response: 200: {"kind":"JobList","apiVersion":"batch/v1","metadata":{"resourceVersion":"113363"},"items":[]} builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [buildman.manager.ephemeral] Starting builder for job buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8 with selected executor: openshift builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [buildman.manager.executor] Using Kubernetes Distribution: openshift builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [buildman.manager.executor] Generated kubernetes resource: builder stdout | {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'namespace': 'virtual-builds', 'generateName': '582cef57-43a5-43d1-9ca7-1d229e4a10d8-', 'labels': {'build': '582cef57-43a5-43d1-9ca7-1d229e4a10d8', 'time': '2023-01-17-06', 'manager': 'quayregistry-quay-app-56b949c797-w2k29', 'quay-sha': 'none'}}, 'spec': {'activeDeadlineSeconds': 7200, 'backoffLimit': 1, 'template': {'metadata': {'labels': {'build': '582cef57-43a5-43d1-9ca7-1d229e4a10d8', 'time': '2023-01-17-06', 'manager': 'quayregistry-quay-app-56b949c797-w2k29', 'quay-sha': 'none'}}, 'spec': {'imagePullSecrets': [{'name': 'builder'}], 'restartPolicy': 'Never', 'dnsPolicy': 'Default', 'containers': [{'name': 'builder', 'imagePullPolicy': 'Always', 'image': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.1', 'env': [{'name': 'TOKEN', 'value': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkRNWEQ0NFI4dXVkNkJHVDJMTzE2TDRmWGtweTlrdmFuM192TngyY0IzbGsifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTFhejIyLnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE2NzM5MzY2MTcsImlhdCI6MTY3MzkzNjYxNywiZXhwIjoxNjczOTM2Njc3LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjU4MmNlZjU3LTQzYTUtNDNkMS05Y2E3LTFkMjI5ZTRhMTBkOCIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvNTgyY2VmNTctNDNhNS00M2QxLTljYTctMWQyMjllNGExMGQ4IiwiZXhwaXJhdGlvbiI6NjB9fQ.PTaB5qFUE-Kih-GTEgULQQQ7CnyCt_x-ivdBSQkS-otzqpfIt_yumqbnfVyFRgO5VfqSqU9g-Rg6qKb6-Jn2Yg2b3NcFmjNCPVdo_LVhjvEmLHSXaK2FehTn7v87tCaE1QEMd7lzqo1bYpKuwFCvTUFeYL6-cupDF22XaEz6EbqaztWHZBv5OH0IeFARmfKaEQQJ0BF1Ewv99L24yqKn3pAMmxN9RHwNNOtNN1Ck_Cr_1YgAZXzHToCbdw5N6LitlC26HxbyN1ZDaGnFZGVVq7xNCLtlNlBCB2Jpn9HlUiEKv9u9ujjg05CGv2Nwm3mjsWomxcSgMzTAytkaMFVQ0w'}, {'name': 'BUILD_UUID', 'value': '582cef57-43a5-43d1-9ca7-1d229e4a10d8'}, {'name': 'SERVER', 'value': 'quayregistry-quay-builder-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443'}, {'name': 'REGISTRY_HOSTNAME', 'value': 'quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com'}, {'name': 'CONTAINER_RUNTIME', 'value': 'podman'}, {'name': 'BULDAH_ISOLATION', 'value': 'chroot'}, {'name': 'CA_CERT', 'value': '-----BEGIN CERTIFICATE-----\nMIIE2DCCA8CgAwIBAgIUOQ8chLyHUIKELwYtqVyZckXsBgIwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTIzMDExNzAzMDk1MFoXDTI0MDEwODAzMDk1MFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTFhejIyLnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALnB6DG2ve/5\ncA68xkTUVE29C1xO6ce68TIxhGpWU28pbViZUsOeTQfgoHU5SWQUHi9qtNqHxcJL\nI5LlWGEqjWANtQhWPp4lupWSkkfEF/iEgREa41/TnVrRdEoh6eU/CQTAhx4G7bMH\nW05rKktUSuSVK62bjzfh56whIBZdJTGemnWYm03YltFHpK25tvjqozpXJOuRv7sx\nDOQrpp+yDekargTilgfnMukXe4QKW00myvbxhBmgTNt42A+zhD96ZEnSaTFsAPaD\nI59P3BblrfxeN66EMYlQFf37ScT0fdOP/84/TPIOPEY6WbysV0WErMWwRMzLJDM/\nBD/+PHaVha8CAwEAAaOCARcwggETMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDExYXoyMi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDExYXoyMi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wDQYJKoZIhvcNAQELBQADggEBAGpCF7EBNrZ3o5tLdoqasi/mw6F0\nk64Qroj6icRSsa+FMqPeHNnOfvtF9rlJHXhjs0KEGuI9L/uUKuvisIeU+uXp2qV7\nRFfBwV3Nfl62DsOC19rOeexljpYyH4f57FwUf0TtULscLGakMxQvkM+lpcvpc6Fc\nXjRliKCav0fqCK8UXdBTHpktp9tnUk22GzoV+jn1IWYIBokmJevwXDUDkHrwmeKy\nehVZVxHXa6TuwkG9XhbACUnKVzUpfS+Ys4cOpVCjgNPTfR3sq/9Qpx2ulssZSHSo\nVQYe2nt0hLDFU11Vk7pAhQNH6oS8bzOsubFRDpDCnyVoJYy5B6n15lNzc78=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUO3mcTeBwB2bwQU38xT2pURipBAAwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTIzMDExNzAzMDk1MFoXDTI1MTEwNjAzMDk1MFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv8kHJFd+GXGX47Oj1yF5\nYNrtaOYCMZGpGACqUfIPmDyiVD38yJbtGtf1utLmQvVzaAQ0Brg16af2edrC217h\nrzCgI2UcpZq9cEDm+baiiTenD2VcAbbF0g973bwT3h7NwaIWZB41nwuVYd9soTuI\nfS/tWRd8ahQNT4IKdClIFHk2mMxpN0Q/GHUELpcZCjGGbKVhDxM6MS4YiN77I06e\nG5tZk52EPdd3zo5y1z9vV5x68qycQbazT7xlOJERZfpkLn3PpFOshWzlec3Sa04G\nVXLjJyzGcB/yRWeo3SzN5VSTYEAi4qT4NOrsxWe3RwQxjVyqph2kVDOe4+Q6OrB1\nbwIDAQABo1MwUTAdBgNVHQ4EFgQUnnTNQ2d1+2LZ+FDbFUOdu3ijZDowHwYDVR0j\nBBgwFoAUnnTNQ2d1+2LZ+FDbFUOdu3ijZDowDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAcpSVyEaq8b1ALtyJwIzIbBhjkXxLA9MC/TaUrHQXbKoc\n2gYcC/hn4yjyRagaWqRAoG60mOH1pzDCq46usu0UGUO9yPhGqq1Pf8q+Zv/kvqkY\nhMwz3yV8zItpPswB+npAVfjVlUgHBIiCzZrahHaUlATqvVE4xuTlH8Gj/mZJRf1j\nJeMlEoIZsc6nAfH93fSYeZ+w2dsZvAUaxzmEawnfsrCGwhKNnwoUR86C9I/lf2Yu\nGuZ1+2uYn3yQHnXAozq0J95rRNLbZTfMpgD0DWTTyYk5Bsu7ftsxj9toMca6mG06\n4ysdtJudyTW0agSCat0UVH/2nTUyKHlvImkmta35vQ==\n-----END CERTIFICATE-----\n'}, {'name': 'GIT_SSL_CAINFO', 'value': '/certs/cacert.crt'}, {'name': 'TLS_CERT_PATH', 'value': '/certs/cacert.crt'}, {'name': 'SSL_CERT_FILE', 'value': '/certs/cacert.crt'}, {'name': 'DEBUG', 'value': 'true'}, {'name': 'HTTP_PROXY', 'value': ''}, {'name': 'HTTPS_PROXY', 'value': ''}, {'name': 'NO_PROXY', 'value': ''}, {'name': 'DOCKER_HOST', 'value': 'unix:///tmp/podman-run-1000/podman/podman.sock'}, {'name': 'EXECUTOR', 'value': 'kubernetesPodman'}], 'resources': {'requests': {'memory': '1G', 'cpu': '500m'}, 'limits': {'memory': '1G', 'cpu': '1000m'}}}], 'automountServiceAccountToken': False, 'serviceAccount': 'quay-builder', 'enableServiceLinks': False}}}} builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [buildman.manager.executor] Using service account token for Kubernetes authentication builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [buildman.manager.executor] Executor config: {'BUILDER_CONTAINER_IMAGE': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.1', 'BUILDER_NAMESPACE': 'virtual-builds', 'CONTAINER_CPU_LIMITS': '1000m', 'CONTAINER_CPU_REQUEST': '500m', 'CONTAINER_MEMORY_LIMITS': '1G', 'CONTAINER_MEMORY_REQUEST': '1G', 'DEBUG': True, 'EXECUTOR': 'kubernetesPodman', 'K8S_API_SERVER': 'api.whu411az22.qe.azure.devcluster.openshift.com:6443', 'K8S_API_TLS_CA': '/conf/stack/extra_ca_certs/build_cluster.crt', 'KUBERNETES_DISTRIBUTION': 'openshift', 'NAME': 'openshift', 'NODE_SELECTOR_LABEL_KEY': '', 'NODE_SELECTOR_LABEL_VALUE': '', 'QUAY_PASSWORD': 'eyJhbGciOiJSUzUxMiJ9.eyJzdWIiOiI3YTE0OWIwNWZkZTI0ZWMwYWQ3ZTIwMjQwZmMyYmU2MCJ9.W-4araN4Cus25Nl39ZOp5MmTZBzeJQlGVhaxh1bRWX2hcAhCaoflBf4l2U71ewD16F9My20HPSSpqIJJ1zydnrXKA731WRknwHYetVZ929_vVNUtHcbCvDYTzCseg781DihDbiWm2pK6mbUKLUEXJXzVaAo7hTbXQsgs63EpKGF3bJyN0M76ZnU8BNkMkaj_cWN0ir8As1vf_OEJudIiSmxxUruF-VMANbTkE50UlP-oh2VCqypBxQ81R5VLu4YsdXfNUwl5jdl14T9MGK0OTw7WAt5VdHBQqr3lsjd4WDWJqigemyrdYM5BC1XZXh8j8v_CfU-yaKXNugTtr8LYk1n6Wso1ThxL9eDUfYFvLRYLoJYizGfOjPcoPJugRIh-tS6ki7zy33-Ya4pkSLLX-OdN4Vq1UJvfE13X8PAOSAbWKF7-7dk1DG2LNW1M2UtEelmyVg6KQQ7cRlmMXp7QouaKO9vDKmudDH_2H9ZVrsXhSFTNmoS_YVkDbxqTs6ugtx5TLxbKYYdNnUygDVKprNdRZSZc-adH4JPuRBroA6CTup13PyW92EtePRN6G2IpXInMC9kkApur5qUynddIscb5KnCaYuKNeqQBCQDI7YxP6CCAfm1obf1nUWbDc2uk4DSNitRZJ84hsLBCLTABTyR0aCGYszCeU4zQ-rn5cc8', 'QUAY_USERNAME': '|76db245a-0d75-11e7-a544-28d244ea5a6d.j9is.751956', 'SERVICE_ACCOUNT_NAME': 'quay-builder', 'SERVICE_ACCOUNT_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6ImJFWkNENWJMMFZKVXdYcW8wVUJmUDktNWxiMGI2MHhBcG5vNElQZXZ3WWMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY3NDAxNzMxOCwiaWF0IjoxNjczOTMwOTE4LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjJhYWZmMGYwLWQ0YjctNDBkMC05NTYxLWU2OWVmNWFjNDAzMCJ9fSwibmJmIjoxNjczOTMwOTE4LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.rxgztiMRRX17HbDGwXvCOAFLYwuIFIoj_0IHKn9Rjvku5WcW76NZFLJ8LkTAeg4mW56vxDmsE-Vvz2a60PhsZbiAQRGmHcYYo36ErCJ-OzTJ7g7qkUx-arCxZjX-CjDI7DlgkC_8BepdpfonxaKiz9NQLH_-6qTu9GOIzxPLueHUfGDIPYqFJQUJkz6c1fxiyUR2J2L0KGEmsPPX2WzwrG0uyUvnup_6nShOoAfXUYAB8L3EXNrKEQaa26FWkuJzDwwg-x50socav-KI68gHAd_ptlyoyUWTFmrRKPHeruKntFC0skPZ7swnwToRKsqGPNEr9dcIxy9kVoEXJUMGV1c9kMN6zFFrBGkBZHo0jv4Ju6sHqYYj8TwnYnax_y_SobBxJ01MGcDIEJX3A60EYiy1_aC5gBm3_uGtpPWGbBqWLzacUcyhKWXfajWNFN_3HHyRu6M5nuQbA3Y_m3R283kaCcsuMMPfGzLQc8lO7vlyyIh4M6VbHpXYVbGExU488jiDC9AoMrXmkLVCmlim0V9oNOx_ZrKIzF53iHVogBATa4n3gYVuDNyK1ywUiM7KNp375vTMYdKdtS4Qen_T838AZuutlVdNeDNSdEHfzEYVLJ42Qky3uqbiAh_7aXaXNK0-Yjp3Df_myEj4M25s7MaUdmU3vfUwEnn40fmM1Ks', 'SETUP_TIME': 180, 'VOLUME_SIZE': '8G'} builder stdout | 2023-01-17 06:23:37,967 [60] [DEBUG] [buildman.manager.executor] Kubernetes request: POST https://api.whu411az22.qe.azure.devcluster.openshift.com:6443/apis/batch/v1/namespaces/virtual-builds/jobs: {'json': {'apiVersion': 'batch/v1', 'kind': 'Job', 'metadata': {'namespace': 'virtual-builds', 'generateName': '582cef57-43a5-43d1-9ca7-1d229e4a10d8-', 'labels': {'build': '582cef57-43a5-43d1-9ca7-1d229e4a10d8', 'time': '2023-01-17-06', 'manager': 'quayregistry-quay-app-56b949c797-w2k29', 'quay-sha': 'none'}}, 'spec': {'activeDeadlineSeconds': 7200, 'backoffLimit': 1, 'template': {'metadata': {'labels': {'build': '582cef57-43a5-43d1-9ca7-1d229e4a10d8', 'time': '2023-01-17-06', 'manager': 'quayregistry-quay-app-56b949c797-w2k29', 'quay-sha': 'none'}}, 'spec': {'imagePullSecrets': [{'name': 'builder'}], 'restartPolicy': 'Never', 'dnsPolicy': 'Default', 'containers': [{'name': 'builder', 'imagePullPolicy': 'Always', 'image': 'brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.1', 'env': [{'name': 'TOKEN', 'value': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkRNWEQ0NFI4dXVkNkJHVDJMTzE2TDRmWGtweTlrdmFuM192TngyY0IzbGsifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTFhejIyLnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE2NzM5MzY2MTcsImlhdCI6MTY3MzkzNjYxNywiZXhwIjoxNjczOTM2Njc3LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjU4MmNlZjU3LTQzYTUtNDNkMS05Y2E3LTFkMjI5ZTRhMTBkOCIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvNTgyY2VmNTctNDNhNS00M2QxLTljYTctMWQyMjllNGExMGQ4IiwiZXhwaXJhdGlvbiI6NjB9fQ.PTaB5qFUE-Kih-GTEgULQQQ7CnyCt_x-ivdBSQkS-otzqpfIt_yumqbnfVyFRgO5VfqSqU9g-Rg6qKb6-Jn2Yg2b3NcFmjNCPVdo_LVhjvEmLHSXaK2FehTn7v87tCaE1QEMd7lzqo1bYpKuwFCvTUFeYL6-cupDF22XaEz6EbqaztWHZBv5OH0IeFARmfKaEQQJ0BF1Ewv99L24yqKn3pAMmxN9RHwNNOtNN1Ck_Cr_1YgAZXzHToCbdw5N6LitlC26HxbyN1ZDaGnFZGVVq7xNCLtlNlBCB2Jpn9HlUiEKv9u9ujjg05CGv2Nwm3mjsWomxcSgMzTAytkaMFVQ0w'}, {'name': 'BUILD_UUID', 'value': '582cef57-43a5-43d1-9ca7-1d229e4a10d8'}, {'name': 'SERVER', 'value': 'quayregistry-quay-builder-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443'}, {'name': 'REGISTRY_HOSTNAME', 'value': 'quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com'}, {'name': 'CONTAINER_RUNTIME', 'value': 'podman'}, {'name': 'BULDAH_ISOLATION', 'value': 'chroot'}, {'name': 'CA_CERT', 'value': '-----BEGIN CERTIFICATE-----\nMIIE2DCCA8CgAwIBAgIUOQ8chLyHUIKELwYtqVyZckXsBgIwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTIzMDExNzAzMDk1MFoXDTI0MDEwODAzMDk1MFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTFhejIyLnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALnB6DG2ve/5\ncA68xkTUVE29C1xO6ce68TIxhGpWU28pbViZUsOeTQfgoHU5SWQUHi9qtNqHxcJL\nI5LlWGEqjWANtQhWPp4lupWSkkfEF/iEgREa41/TnVrRdEoh6eU/CQTAhx4G7bMH\nW05rKktUSuSVK62bjzfh56whIBZdJTGemnWYm03YltFHpK25tvjqozpXJOuRv7sx\nDOQrpp+yDekargTilgfnMukXe4QKW00myvbxhBmgTNt42A+zhD96ZEnSaTFsAPaD\nI59P3BblrfxeN66EMYlQFf37ScT0fdOP/84/TPIOPEY6WbysV0WErMWwRMzLJDM/\nBD/+PHaVha8CAwEAAaOCARcwggETMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDExYXoyMi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDExYXoyMi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wDQYJKoZIhvcNAQELBQADggEBAGpCF7EBNrZ3o5tLdoqasi/mw6F0\nk64Qroj6icRSsa+FMqPeHNnOfvtF9rlJHXhjs0KEGuI9L/uUKuvisIeU+uXp2qV7\nRFfBwV3Nfl62DsOC19rOeexljpYyH4f57FwUf0TtULscLGakMxQvkM+lpcvpc6Fc\nXjRliKCav0fqCK8UXdBTHpktp9tnUk22GzoV+jn1IWYIBokmJevwXDUDkHrwmeKy\nehVZVxHXa6TuwkG9XhbACUnKVzUpfS+Ys4cOpVCjgNPTfR3sq/9Qpx2ulssZSHSo\nVQYe2nt0hLDFU11Vk7pAhQNH6oS8bzOsubFRDpDCnyVoJYy5B6n15lNzc78=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUO3mcTeBwB2bwQU38xT2pURipBAAwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTIzMDExNzAzMDk1MFoXDTI1MTEwNjAzMDk1MFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv8kHJFd+GXGX47Oj1yF5\nYNrtaOYCMZGpGACqUfIPmDyiVD38yJbtGtf1utLmQvVzaAQ0Brg16af2edrC217h\nrzCgI2UcpZq9cEDm+baiiTenD2VcAbbF0g973bwT3h7NwaIWZB41nwuVYd9soTuI\nfS/tWRd8ahQNT4IKdClIFHk2mMxpN0Q/GHUELpcZCjGGbKVhDxM6MS4YiN77I06e\nG5tZk52EPdd3zo5y1z9vV5x68qycQbazT7xlOJERZfpkLn3PpFOshWzlec3Sa04G\nVXLjJyzGcB/yRWeo3SzN5VSTYEAi4qT4NOrsxWe3RwQxjVyqph2kVDOe4+Q6OrB1\nbwIDAQABo1MwUTAdBgNVHQ4EFgQUnnTNQ2d1+2LZ+FDbFUOdu3ijZDowHwYDVR0j\nBBgwFoAUnnTNQ2d1+2LZ+FDbFUOdu3ijZDowDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAcpSVyEaq8b1ALtyJwIzIbBhjkXxLA9MC/TaUrHQXbKoc\n2gYcC/hn4yjyRagaWqRAoG60mOH1pzDCq46usu0UGUO9yPhGqq1Pf8q+Zv/kvqkY\nhMwz3yV8zItpPswB+npAVfjVlUgHBIiCzZrahHaUlATqvVE4xuTlH8Gj/mZJRf1j\nJeMlEoIZsc6nAfH93fSYeZ+w2dsZvAUaxzmEawnfsrCGwhKNnwoUR86C9I/lf2Yu\nGuZ1+2uYn3yQHnXAozq0J95rRNLbZTfMpgD0DWTTyYk5Bsu7ftsxj9toMca6mG06\n4ysdtJudyTW0agSCat0UVH/2nTUyKHlvImkmta35vQ==\n-----END CERTIFICATE-----\n'}, {'name': 'GIT_SSL_CAINFO', 'value': '/certs/cacert.crt'}, {'name': 'TLS_CERT_PATH', 'value': '/certs/cacert.crt'}, {'name': 'SSL_CERT_FILE', 'value': '/certs/cacert.crt'}, {'name': 'DEBUG', 'value': 'true'}, {'name': 'HTTP_PROXY', 'value': ''}, {'name': 'HTTPS_PROXY', 'value': ''}, {'name': 'NO_PROXY', 'value': ''}, {'name': 'DOCKER_HOST', 'value': 'unix:///tmp/podman-run-1000/podman/podman.sock'}, {'name': 'EXECUTOR', 'value': 'kubernetesPodman'}], 'resources': {'requests': {'memory': '1G', 'cpu': '500m'}, 'limits': {'memory': '1G', 'cpu': '1000m'}}}], 'automountServiceAccountToken': False, 'serviceAccount': 'quay-builder', 'enableServiceLinks': False}}}}, 'verify': '/conf/stack/extra_ca_certs/build_cluster.crt', 'timeout': 20, 'headers': {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJFWkNENWJMMFZKVXdYcW8wVUJmUDktNWxiMGI2MHhBcG5vNElQZXZ3WWMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY3NDAxNzMxOCwiaWF0IjoxNjczOTMwOTE4LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InZpcnR1YWwtYnVpbGRzIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InF1YXktYnVpbGRlciIsInVpZCI6IjJhYWZmMGYwLWQ0YjctNDBkMC05NTYxLWU2OWVmNWFjNDAzMCJ9fSwibmJmIjoxNjczOTMwOTE4LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dmlydHVhbC1idWlsZHM6cXVheS1idWlsZGVyIn0.rxgztiMRRX17HbDGwXvCOAFLYwuIFIoj_0IHKn9Rjvku5WcW76NZFLJ8LkTAeg4mW56vxDmsE-Vvz2a60PhsZbiAQRGmHcYYo36ErCJ-OzTJ7g7qkUx-arCxZjX-CjDI7DlgkC_8BepdpfonxaKiz9NQLH_-6qTu9GOIzxPLueHUfGDIPYqFJQUJkz6c1fxiyUR2J2L0KGEmsPPX2WzwrG0uyUvnup_6nShOoAfXUYAB8L3EXNrKEQaa26FWkuJzDwwg-x50socav-KI68gHAd_ptlyoyUWTFmrRKPHeruKntFC0skPZ7swnwToRKsqGPNEr9dcIxy9kVoEXJUMGV1c9kMN6zFFrBGkBZHo0jv4Ju6sHqYYj8TwnYnax_y_SobBxJ01MGcDIEJX3A60EYiy1_aC5gBm3_uGtpPWGbBqWLzacUcyhKWXfajWNFN_3HHyRu6M5nuQbA3Y_m3R283kaCcsuMMPfGzLQc8lO7vlyyIh4M6VbHpXYVbGExU488jiDC9AoMrXmkLVCmlim0V9oNOx_ZrKIzF53iHVogBATa4n3gYVuDNyK1ywUiM7KNp375vTMYdKdtS4Qen_T838AZuutlVdNeDNSdEHfzEYVLJ42Qky3uqbiAh_7aXaXNK0-Yjp3Df_myEj4M25s7MaUdmU3vfUwEnn40fmM1Ks'}} builder stdout | 2023-01-17 06:23:37,969 [60] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.whu411az22.qe.azure.devcluster.openshift.com:6443 builder stdout | 2023-01-17 06:23:37,985 [60] [DEBUG] [urllib3.connectionpool] https://api.whu411az22.qe.azure.devcluster.openshift.com:6443 "POST /apis/batch/v1/namespaces/virtual-builds/jobs HTTP/1.1" 201 None builder stdout | 2023-01-17 06:23:37,985 [60] [DEBUG] [buildman.manager.executor] Kubernetes response: 201: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk","generateName":"582cef57-43a5-43d1-9ca7-1d229e4a10d8-","namespace":"virtual-builds","uid":"6be66b3c-e580-4c69-b960-148a40ba5c7e","resourceVersion":"113364","generation":1,"creationTimestamp":"2023-01-17T06:23:37Z","labels":{"build":"582cef57-43a5-43d1-9ca7-1d229e4a10d8","manager":"quayregistry-quay-app-56b949c797-w2k29","quay-sha":"none","time":"2023-01-17-06"},"managedFields":[{"manager":"python-requests","operation":"Update","apiVersion":"batch/v1","time":"2023-01-17T06:23:37Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:activeDeadlineSeconds":{},"f:backoffLimit":{},"f:completionMode":{},"f:completions":{},"f:parallelism":{},"f:suspend":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:build":{},"f:manager":{},"f:quay-sha":{},"f:time":{}}},"f:spec":{"f:automountServiceAccountToken":{},"f:containers":{"k:{\"name\":\"builder\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BUILD_UUID\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BULDAH_ISOLATION\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CA_CERT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_RUNTIME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DEBUG\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DOCKER_HOST\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"EXECUTOR\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"GIT_SSL_CAINFO\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"HTTPS_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"HTTP_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"NO_PROXY\"}":{".":{},"f:name":{}},"k:{\"name\":\"REGISTRY_HOSTNAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SERVER\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SSL_CERT_FILE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TLS_CERT_PATH\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"TOKEN\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:imagePullSecrets":{".":{},"k:{\"name\":\"builder\"}":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}}}}]},"spec":{"parallelism":1,"completions":1,"activeDeadlineSeconds":7200,"backoffLimit":1,"selector":{"matchLabels":{"controller-uid":"6be66b3c-e580-4c69-b960-148a40ba5c7e"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"build":"582cef57-43a5-43d1-9ca7-1d229e4a10d8","controller-uid":"6be66b3c-e580-4c69-b960-148a40ba5c7e","job-name":"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk","manager":"quayregistry-quay-app-56b949c797-w2k29","quay-sha":"none","time":"2023-01-17-06"}},"spec":{"containers":[{"name":"builder","image":"brew.registry.redhat.io/rh-osbs/quay-quay-builder-rhel8:v3.8.1","env":[{"name":"TOKEN","value":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkRNWEQ0NFI4dXVkNkJHVDJMTzE2TDRmWGtweTlrdmFuM192TngyY0IzbGsifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktYnVpbGRlci1xdWF5LWVudGVycHJpc2UuYXBwcy53aHU0MTFhejIyLnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNoaWZ0LmNvbTo0NDMiLCJuYmYiOjE2NzM5MzY2MTcsImlhdCI6MTY3MzkzNjYxNywiZXhwIjoxNjczOTM2Njc3LCJzdWIiOiIoYW5vbnltb3VzKSIsImFjY2VzcyI6e30sImNvbnRleHQiOnsidG9rZW5fdHlwZSI6ImJ1aWxkX2pvYl9yZWdpc3RyYXRpb24iLCJidWlsZF9pZCI6IjU4MmNlZjU3LTQzYTUtNDNkMS05Y2E3LTFkMjI5ZTRhMTBkOCIsImpvYl9pZCI6ImJ1aWxkbWFuL3Byb2R1Y3Rpb24vYnVpbGRpbmcvNTgyY2VmNTctNDNhNS00M2QxLTljYTctMWQyMjllNGExMGQ4IiwiZXhwaXJhdGlvbiI6NjB9fQ.PTaB5qFUE-Kih-GTEgULQQQ7CnyCt_x-ivdBSQkS-otzqpfIt_yumqbnfVyFRgO5VfqSqU9g-Rg6qKb6-Jn2Yg2b3NcFmjNCPVdo_LVhjvEmLHSXaK2FehTn7v87tCaE1QEMd7lzqo1bYpKuwFCvTUFeYL6-cupDF22XaEz6EbqaztWHZBv5OH0IeFARmfKaEQQJ0BF1Ewv99L24yqKn3pAMmxN9RHwNNOtNN1Ck_Cr_1YgAZXzHToCbdw5N6LitlC26HxbyN1ZDaGnFZGVVq7xNCLtlNlBCB2Jpn9HlUiEKv9u9ujjg05CGv2Nwm3mjsWomxcSgMzTAytkaMFVQ0w"},{"name":"BUILD_UUID","value":"582cef57-43a5-43d1-9ca7-1d229e4a10d8"},{"name":"SERVER","value":"quayregistry-quay-builder-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443"},{"name":"REGISTRY_HOSTNAME","value":"quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com"},{"name":"CONTAINER_RUNTIME","value":"podman"},{"name":"BULDAH_ISOLATION","value":"chroot"},{"name":"CA_CERT","value":"-----BEGIN CERTIFICATE-----\nMIIE2DCCA8CgAwIBAgIUOQ8chLyHUIKELwYtqVyZckXsBgIwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTIzMDExNzAzMDk1MFoXDTI0MDEwODAzMDk1MFowgZsxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xQjBABgNVBAMMOXdo\ndXF1YXkuYXBwcy53aHU0MTFhejIyLnFlLmF6dXJlLmRldmNsdXN0ZXIub3BlbnNo\naWZ0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALnB6DG2ve/5\ncA68xkTUVE29C1xO6ce68TIxhGpWU28pbViZUsOeTQfgoHU5SWQUHi9qtNqHxcJL\nI5LlWGEqjWANtQhWPp4lupWSkkfEF/iEgREa41/TnVrRdEoh6eU/CQTAhx4G7bMH\nW05rKktUSuSVK62bjzfh56whIBZdJTGemnWYm03YltFHpK25tvjqozpXJOuRv7sx\nDOQrpp+yDekargTilgfnMukXe4QKW00myvbxhBmgTNt42A+zhD96ZEnSaTFsAPaD\nI59P3BblrfxeN66EMYlQFf37ScT0fdOP/84/TPIOPEY6WbysV0WErMWwRMzLJDM/\nBD/+PHaVha8CAwEAAaOCARcwggETMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMIH4\nBgNVHREEgfAwge2CW3F1YXlyZWdpc3RyeS1xdWF5LWJ1aWxkZXItcXVheS1lbnRl\ncnByaXNlLmFwcHMud2h1NDExYXoyMi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb22CU3F1YXlyZWdpc3RyeS1xdWF5LXF1YXktZW50ZXJwcmlzZS5hcHBz\nLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tgjl3\naHVxdWF5LmFwcHMud2h1NDExYXoyMi5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5z\naGlmdC5jb20wDQYJKoZIhvcNAQELBQADggEBAGpCF7EBNrZ3o5tLdoqasi/mw6F0\nk64Qroj6icRSsa+FMqPeHNnOfvtF9rlJHXhjs0KEGuI9L/uUKuvisIeU+uXp2qV7\nRFfBwV3Nfl62DsOC19rOeexljpYyH4f57FwUf0TtULscLGakMxQvkM+lpcvpc6Fc\nXjRliKCav0fqCK8UXdBTHpktp9tnUk22GzoV+jn1IWYIBokmJevwXDUDkHrwmeKy\nehVZVxHXa6TuwkG9XhbACUnKVzUpfS+Ys4cOpVCjgNPTfR3sq/9Qpx2ulssZSHSo\nVQYe2nt0hLDFU11Vk7pAhQNH6oS8bzOsubFRDpDCnyVoJYy5B6n15lNzc78=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIECzCCAvOgAwIBAgIUO3mcTeBwB2bwQU38xT2pURipBAAwDQYJKoZIhvcNAQEL\nBQAwgZQxCzAJBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJC\nSjESMBAGA1UECgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5\nBgNVBAMMMi5hcHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVu\nc2hpZnQuY29tMB4XDTIzMDExNzAzMDk1MFoXDTI1MTEwNjAzMDk1MFowgZQxCzAJ\nBgNVBAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMQswCQYDVQQHDAJCSjESMBAGA1UE\nCgwJUXVheSB0ZWFtMRUwEwYDVQQLDAxRdWF5IFFFIFRlYW0xOzA5BgNVBAMMMi5h\ncHBzLndodTQxMWF6MjIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29t\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv8kHJFd+GXGX47Oj1yF5\nYNrtaOYCMZGpGACqUfIPmDyiVD38yJbtGtf1utLmQvVzaAQ0Brg16af2edrC217h\nrzCgI2UcpZq9cEDm+baiiTenD2VcAbbF0g973bwT3h7NwaIWZB41nwuVYd9soTuI\nfS/tWRd8ahQNT4IKdClIFHk2mMxpN0Q/GHUELpcZCjGGbKVhDxM6MS4YiN77I06e\nG5tZk52EPdd3zo5y1z9vV5x68qycQbazT7xlOJERZfpkLn3PpFOshWzlec3Sa04G\nVXLjJyzGcB/yRWeo3SzN5VSTYEAi4qT4NOrsxWe3RwQxjVyqph2kVDOe4+Q6OrB1\nbwIDAQABo1MwUTAdBgNVHQ4EFgQUnnTNQ2d1+2LZ+FDbFUOdu3ijZDowHwYDVR0j\nBBgwFoAUnnTNQ2d1+2LZ+FDbFUOdu3ijZDowDwYDVR0TAQH/BAUwAwEB/zANBgkq\nhkiG9w0BAQsFAAOCAQEAcpSVyEaq8b1ALtyJwIzIbBhjkXxLA9MC/TaUrHQXbKoc\n2gYcC/hn4yjyRagaWqRAoG60mOH1pzDCq46usu0UGUO9yPhGqq1Pf8q+Zv/kvqkY\nhMwz3yV8zItpPswB+npAVfjVlUgHBIiCzZrahHaUlATqvVE4xuTlH8Gj/mZJRf1j\nJeMlEoIZsc6nAfH93fSYeZ+w2dsZvAUaxzmEawnfsrCGwhKNnwoUR86C9I/lf2Yu\nGuZ1+2uYn3yQHnXAozq0J95rRNLbZTfMpgD0DWTTyYk5Bsu7ftsxj9toMca6mG06\n4ysdtJudyTW0agSCat0UVH/2nTUyKHlvImkmta35vQ==\n-----END CERTIFICATE-----\n"},{"name":"GIT_SSL_CAINFO","value":"/certs/cacert.crt"},{"name":"TLS_CERT_PATH","value":"/certs/cacert.crt"},{"name":"SSL_CERT_FILE","value":"/certs/cacert.crt"},{"name":"DEBUG","value":"true"},{"name":"HTTP_PROXY"},{"name":"HTTPS_PROXY"},{"name":"NO_PROXY"},{"name":"DOCKER_HOST","value":"unix:///tmp/podman-run-1000/podman/podman.sock"},{"name":"EXECUTOR","value":"kubernetesPodman"}],"resources":{"limits":{"cpu":"1","memory":"1G"},"requests":{"cpu":"500m","memory":"1G"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"Default","serviceAccountName":"quay-builder","serviceAccount":"quay-builder","automountServiceAccountToken":false,"securityContext":{},"imagePullSecrets":[{"name":"builder"}],"schedulerName":"default-scheduler","enableServiceLinks":false}},"completionMode":"NonIndexed","suspend":false},"status":{}} builder stdout | 2023-01-17 06:23:37,986 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": null, \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:23:37,997 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:23:38,006 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:23:38,014 [60] [DEBUG] [buildman.manager.ephemeral] Job scheduled for job buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8 with execution with ID 582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk on control plane openshift with max startup time of 180 builder stdout | 2023-01-17 06:23:38,015 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 1, "key": "buildman/production/metric/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"executor_name\": \"openshift\", \"start_time\": 1673936617.9860892}"} builder stdout | 2023-01-17 06:23:38,015 [60] [DEBUG] [buildman.manager.ephemeral] Build job buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8 scheduled. builder stdout | 2023-01-17 06:23:38,015 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:38,021 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:23:38,716 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:23:38,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:38,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:43.935629+00:00 (in 5.200175 seconds) notificationworker stdout | 2023-01-17 06:23:38,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:48 UTC)" (scheduled at 2023-01-17 06:23:38.735066+00:00) notificationworker stdout | 2023-01-17 06:23:38,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:38,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 38, 735850), True, datetime.datetime(2023, 1, 17, 6, 23, 38, 735850), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:38,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:38,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:38,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:48 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:23:39,031 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:23:39,031 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:48.738458+00:00 (in 9.707186 seconds) repositorygcworker stdout | 2023-01-17 06:23:39,031 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:39 UTC)" (scheduled at 2023-01-17 06:23:39.030814+00:00) repositorygcworker stdout | 2023-01-17 06:23:39,031 [85] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:23:39,032 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 39, 31703), True, datetime.datetime(2023, 1, 17, 6, 23, 39, 31703), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:23:39,039 [85] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:23:39,039 [85] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:23:39,040 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:39 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:39,198 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:23:39,202 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:23:39,209 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:23:39,212 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:23:39,215 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-web stdout | 2023-01-17 06:23:39,218 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:23:39,220 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:23:39,224 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} securityworker stdout | 2023-01-17 06:23:39,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:39,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:52.837365+00:00 (in 13.606919 seconds) securityworker stdout | 2023-01-17 06:23:39,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:09 UTC)" (scheduled at 2023-01-17 06:23:39.230038+00:00) securityworker stdout | 2023-01-17 06:23:39,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:39,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:23:39,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:39,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:23:39,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:39,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:39,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state gunicorn-web stdout | 2023-01-17 06:23:39,233 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:23:39,233 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} securityworker stdout | 2023-01-17 06:23:39,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:39,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:23:39,236 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} gunicorn-registry stdout | 2023-01-17 06:23:39,239 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} securityworker stdout | 2023-01-17 06:23:39,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:39,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:23:39,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:23:39,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:23:39,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:39,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:39,242 [87] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 06:23:39,242 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} securityworker stdout | 2023-01-17 06:23:39,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:23:39,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:09 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:23:39,244 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} logrotateworker stdout | 2023-01-17 06:23:39,596 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:23:39,618 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:23:40,694 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:23:41,365 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:23:42,161 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:23:42,163 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gcworker stdout | 2023-01-17 06:23:42,184 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 06:23:42,233 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} notificationworker stdout | 2023-01-17 06:23:43,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:43,936 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:48.735066+00:00 (in 4.798975 seconds) notificationworker stdout | 2023-01-17 06:23:43,936 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:43 UTC)" (scheduled at 2023-01-17 06:23:43.935629+00:00) notificationworker stdout | 2023-01-17 06:23:43,936 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:23:43,936 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:43 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:23:47,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:23:47,933 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:59.337413+00:00 (in 11.404475 seconds) namespacegcworker stdout | 2023-01-17 06:23:47,933 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:47 UTC)" (scheduled at 2023-01-17 06:23:47.932490+00:00) namespacegcworker stdout | 2023-01-17 06:23:47,933 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:23:47,933 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:47 UTC)" executed successfully builder stdout | 2023-01-17 06:23:48,028 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:48,037 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:48,037 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:48,043 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:48,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:48,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:56.935055+00:00 (in 8.199434 seconds) notificationworker stdout | 2023-01-17 06:23:48,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:58 UTC)" (scheduled at 2023-01-17 06:23:48.735066+00:00) notificationworker stdout | 2023-01-17 06:23:48,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:48,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 48, 735987), True, datetime.datetime(2023, 1, 17, 6, 23, 48, 735987), 0, 'notification/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:23:48,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:23:48,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:39.030814+00:00 (in 50.291910 seconds) repositorygcworker stdout | 2023-01-17 06:23:48,739 [85] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:48 UTC)" (scheduled at 2023-01-17 06:23:48.738458+00:00) repositorygcworker stdout | 2023-01-17 06:23:48,739 [85] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:23:48,739 [85] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:48 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:23:48,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:48,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:48,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:58 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:23:48,830 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:23:48,830 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:24.641816+00:00 (in 35.811093 seconds) exportactionlogsworker stdout | 2023-01-17 06:23:48,830 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:48 UTC)" (scheduled at 2023-01-17 06:23:48.830274+00:00) exportactionlogsworker stdout | 2023-01-17 06:23:48,831 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:23:48,831 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 48, 831084), True, datetime.datetime(2023, 1, 17, 6, 23, 48, 831084), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:23:48,838 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:23:48,838 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:23:48,839 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:48 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:52,625 [243] [DEBUG] [app] Starting request: urn:request:9aad79d0-e6b1-406d-a2b7-4ed8c73ec72d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:52,625 [244] [DEBUG] [app] Starting request: urn:request:1ef6ee90-0d7d-4a41-be99-cad3fbc2763d (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:23:52,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:52,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:52,630 [232] [DEBUG] [app] Starting request: urn:request:20fdee61-6b20-49c2-a789-26a5295d61bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:52,631 [242] [DEBUG] [app] Starting request: urn:request:fd88ad55-1a75-48f4-a2be-0c3b1ce71908 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:52,631 [232] [DEBUG] [app] Ending request: urn:request:20fdee61-6b20-49c2-a789-26a5295d61bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20fdee61-6b20-49c2-a789-26a5295d61bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:52,631 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:23:52,631 [242] [DEBUG] [app] Ending request: urn:request:fd88ad55-1a75-48f4-a2be-0c3b1ce71908 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd88ad55-1a75-48f4-a2be-0c3b1ce71908', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:52,631 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:23:52,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:23:52,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:52,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:52,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:52,636 [239] [DEBUG] [app] Starting request: urn:request:c74c1826-5417-4323-bc6c-2a4e71faa742 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:52,636 [241] [DEBUG] [app] Starting request: urn:request:ae02c0d9-6044-4735-ac12-dacb92804ed9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:52,636 [239] [DEBUG] [app] Ending request: urn:request:c74c1826-5417-4323-bc6c-2a4e71faa742 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c74c1826-5417-4323-bc6c-2a4e71faa742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:52,636 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:52,636 [241] [DEBUG] [app] Ending request: urn:request:ae02c0d9-6044-4735-ac12-dacb92804ed9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae02c0d9-6044-4735-ac12-dacb92804ed9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:52,636 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:52,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:52,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:52,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:52,637 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:52,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:52,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:52,648 [243] [DEBUG] [app] Ending request: urn:request:9aad79d0-e6b1-406d-a2b7-4ed8c73ec72d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9aad79d0-e6b1-406d-a2b7-4ed8c73ec72d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:52,648 [244] [DEBUG] [app] Ending request: urn:request:1ef6ee90-0d7d-4a41-be99-cad3fbc2763d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ef6ee90-0d7d-4a41-be99-cad3fbc2763d', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:52,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:52,648 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.023 47 0.024) gunicorn-web stdout | 2023-01-17 06:23:52,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:52,648 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:23:52,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:52,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:09.230038+00:00 (in 16.392241 seconds) securityworker stdout | 2023-01-17 06:23:52,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:22 UTC)" (scheduled at 2023-01-17 06:23:52.837365+00:00) securityworker stdout | 2023-01-17 06:23:52,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:52,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:23:52,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:52,842 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:52,848 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:52,850 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:52,850 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:22 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:23:56,935 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:56,935 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:58.735066+00:00 (in 1.799554 seconds) notificationworker stdout | 2023-01-17 06:23:56,935 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:28:56 UTC)" (scheduled at 2023-01-17 06:23:56.935055+00:00) notificationworker stdout | 2023-01-17 06:23:56,936 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 23, 56, 935904), 'notification/%']) notificationworker stdout | 2023-01-17 06:23:56,943 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 23, 56, 935904), True, datetime.datetime(2023, 1, 17, 6, 23, 56, 935904), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:23:56,945 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 23, 56, 935904), True, datetime.datetime(2023, 1, 17, 6, 23, 56, 935904), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 23, 56, 935904), 'notification/%']) notificationworker stdout | 2023-01-17 06:23:56,947 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:56,947 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:28:56 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:23:57,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:23:57,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:27.226424+00:00 (in 29.999554 seconds) buildlogsarchiver stdout | 2023-01-17 06:23:57,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:27 UTC)" (scheduled at 2023-01-17 06:23:57.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:23:57,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 23, 57, 227278), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:23:57,234 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:23:57,234 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:23:57,234 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:27 UTC)" executed successfully builder stdout | 2023-01-17 06:23:58,052 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:58,061 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:58,061 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:58,067 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:58,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:58,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:08.735066+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 06:23:58,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:08 UTC)" (scheduled at 2023-01-17 06:23:58.735066+00:00) notificationworker stdout | 2023-01-17 06:23:58,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:58,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 58, 735983), True, datetime.datetime(2023, 1, 17, 6, 23, 58, 735983), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:58,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:58,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:58,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:08 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:23:59,337 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:23:59,337 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:15.931994+00:00 (in 16.594150 seconds) namespacegcworker stdout | 2023-01-17 06:23:59,338 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:59 UTC)" (scheduled at 2023-01-17 06:23:59.337413+00:00) namespacegcworker stdout | 2023-01-17 06:23:59,338 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:23:59,339 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 59, 338251), True, datetime.datetime(2023, 1, 17, 6, 23, 59, 338251), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:23:59,347 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:23:59,347 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:23:59,347 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:59 UTC)" executed successfully securityworker stdout | 2023-01-17 06:24:03,066 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 06:24:03,429 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} servicekey stdout | 2023-01-17 06:24:03,664 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'servicekeyworker.py', 'pid': '88'} buildlogsarchiver stdout | 2023-01-17 06:24:04,470 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} teamsyncworker stdout | 2023-01-17 06:24:04,569 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'teamsyncworker.py', 'pid': '90'} repositorygcworker stdout | 2023-01-17 06:24:05,737 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositorygcworker.py', 'pid': '85'} gcworker stdout | 2023-01-17 06:24:05,848 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:24:05,849 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:35.848675+00:00 (in 29.999601 seconds) gcworker stdout | 2023-01-17 06:24:05,849 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:35 UTC)" (scheduled at 2023-01-17 06:24:05.848675+00:00) gcworker stdout | 2023-01-17 06:24:05,850 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672727045849, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:24:05,858 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:24:05,858 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:24:05,858 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:35 UTC)" executed successfully builder stdout | 2023-01-17 06:24:06,422 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'builder.py', 'pid': '60'} manifestbackfillworker stdout | 2023-01-17 06:24:06,569 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 06:24:06,676 [215] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '215'} gunicorn-secscan stdout | 2023-01-17 06:24:06,679 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:24:06,681 [216] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'secscan:application', 'pid': '216'} namespacegcworker stdout | 2023-01-17 06:24:06,920 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 06:24:07,625 [244] [DEBUG] [app] Starting request: urn:request:8d5a8e77-ecca-4aac-af4b-73b710692561 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:24:07,625 [243] [DEBUG] [app] Starting request: urn:request:c1662ed6-2ae3-406b-8c1b-ae33f3d18cb0 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:24:07,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,627 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,629 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:07,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:24:07,631 [232] [DEBUG] [app] Starting request: urn:request:5baff8b1-48ef-420d-afbc-4d01af640b38 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:07,631 [242] [DEBUG] [app] Starting request: urn:request:274c8bad-cfbe-4e95-8ed5-4337e79d6e3d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:07,631 [232] [DEBUG] [app] Ending request: urn:request:5baff8b1-48ef-420d-afbc-4d01af640b38 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5baff8b1-48ef-420d-afbc-4d01af640b38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:24:07,632 [242] [DEBUG] [app] Ending request: urn:request:274c8bad-cfbe-4e95-8ed5-4337e79d6e3d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:274c8bad-cfbe-4e95-8ed5-4337e79d6e3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:24:07,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:24:07,632 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:24:07,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:24:07,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:07,633 [243] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,633 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,635 [243] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:07,635 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:07,636 [239] [DEBUG] [app] Starting request: urn:request:804a8634-0fba-440d-966a-847da6809f31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:07,636 [239] [DEBUG] [app] Ending request: urn:request:804a8634-0fba-440d-966a-847da6809f31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:804a8634-0fba-440d-966a-847da6809f31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:07,636 [241] [DEBUG] [app] Starting request: urn:request:852f7a17-7cef-41ec-9115-ae3a4b284087 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:07,637 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:24:07,637 [243] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:07,637 [241] [DEBUG] [app] Ending request: urn:request:852f7a17-7cef-41ec-9115-ae3a4b284087 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:852f7a17-7cef-41ec-9115-ae3a4b284087', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:24:07,637 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:24:07,637 [243] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:07,637 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:07,638 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:07,642 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:07,642 [243] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:07,648 [243] [DEBUG] [app] Ending request: urn:request:c1662ed6-2ae3-406b-8c1b-ae33f3d18cb0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1662ed6-2ae3-406b-8c1b-ae33f3d18cb0', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:07,648 [243] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:24:07,648 [244] [DEBUG] [app] Ending request: urn:request:8d5a8e77-ecca-4aac-af4b-73b710692561 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d5a8e77-ecca-4aac-af4b-73b710692561', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:07,648 [244] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:24:07,648 [243] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:24:07,649 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) repositoryactioncounter stdout | 2023-01-17 06:24:07,921 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'repositoryactioncounter.py', 'pid': '84'} globalpromstats stdout | 2023-01-17 06:24:07,970 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:24:08,077 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:24:08,086 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:24:08,086 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:24:08,092 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:24:08,723 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 06:24:08,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:24:08,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:18.735066+00:00 (in 9.999601 seconds) notificationworker stdout | 2023-01-17 06:24:08,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:18 UTC)" (scheduled at 2023-01-17 06:24:08.735066+00:00) notificationworker stdout | 2023-01-17 06:24:08,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:24:08,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 8, 735850), True, datetime.datetime(2023, 1, 17, 6, 24, 8, 735850), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:24:08,743 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:24:08,743 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:08,743 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:18 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:24:09,205 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:24:09,213 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:24:09,218 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:24:09,221 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:24:09,224 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '244'} gunicorn-registry stdout | 2023-01-17 06:24:09,226 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:24:09,229 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '242'} securityworker stdout | 2023-01-17 06:24:09,230 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:24:09,230 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:22.837365+00:00 (in 13.606902 seconds) securityworker stdout | 2023-01-17 06:24:09,230 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:39 UTC)" (scheduled at 2023-01-17 06:24:09.230038+00:00) securityworker stdout | 2023-01-17 06:24:09,230 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:09,230 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:24:09,231 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:24:09,231 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:24:09,231 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:09,231 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:24:09,231 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state gunicorn-registry stdout | 2023-01-17 06:24:09,233 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '68'} securityworker stdout | 2023-01-17 06:24:09,234 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:24:09,235 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:24:09,239 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '243'} securityworker stdout | 2023-01-17 06:24:09,241 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:09,241 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:24:09,241 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:24:09,241 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:24:09,242 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:24:09,242 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:09,242 [87] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:24:09,242 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 06:24:09,242 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:24:09,242 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:39 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:24:09,246 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:24:09,249 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '235'} gunicorn-registry stdout | 2023-01-17 06:24:09,251 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '234'} gunicorn-registry stdout | 2023-01-17 06:24:09,253 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'registry:application', 'pid': '233'} logrotateworker stdout | 2023-01-17 06:24:09,632 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'logrotateworker.py', 'pid': '71'} queuecleanupworker stdout | 2023-01-17 06:24:09,653 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'queuecleanupworker.py', 'pid': '78'} storagereplication stdout | 2023-01-17 06:24:10,729 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 06:24:11,401 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'notificationworker.py', 'pid': '76'} expiredappspecifictokenworker stdout | 2023-01-17 06:24:12,194 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} blobuploadcleanupworker stdout | 2023-01-17 06:24:12,197 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gcworker stdout | 2023-01-17 06:24:12,217 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'gcworker.py', 'pid': '66'} securityscanningnotificationworker stdout | 2023-01-17 06:24:12,263 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityscanningnotificationworker.py', 'pid': '86'} namespacegcworker stdout | 2023-01-17 06:24:15,932 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:24:15,932 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:47.932490+00:00 (in 32.000056 seconds) namespacegcworker stdout | 2023-01-17 06:24:15,932 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:29:15 UTC)" (scheduled at 2023-01-17 06:24:15.931994+00:00) namespacegcworker stdout | 2023-01-17 06:24:15,933 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 24, 15, 932850), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:24:15,940 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 24, 15, 932850), True, datetime.datetime(2023, 1, 17, 6, 24, 15, 932850), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:24:15,943 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 24, 15, 932850), True, datetime.datetime(2023, 1, 17, 6, 24, 15, 932850), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 24, 15, 932850), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:24:15,945 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:24:15,945 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:29:15 UTC)" executed successfully builder stdout | 2023-01-17 06:24:18,093 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:24:18,102 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:24:18,102 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:24:18,108 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:18,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:24:18,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:28.735066+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 06:24:18,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:28 UTC)" (scheduled at 2023-01-17 06:24:18.735066+00:00) notificationworker stdout | 2023-01-17 06:24:18,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:24:18,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 18, 735978), True, datetime.datetime(2023, 1, 17, 6, 24, 18, 735978), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:24:18,744 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:24:18,744 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:18,744 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:28 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:24:20,242 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:24:20,242 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:25.334138+00:00 (in 5.091507 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:24:20,242 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:20 UTC)" (scheduled at 2023-01-17 06:24:20.242198+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:24:20,243 [86] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:24:20,243 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 20, 243037), True, datetime.datetime(2023, 1, 17, 6, 24, 20, 243037), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:24:20,251 [86] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:24:20,251 [86] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:24:20,251 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:20 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:24:22,625 [239] [DEBUG] [app] Starting request: urn:request:cb84f918-e50a-4771-b10e-06175d187bc1 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:24:22,626 [244] [DEBUG] [app] Starting request: urn:request:6c7e7dad-f6e4-4284-b1cc-9e1afdb1d4a5 (/health/instance) {'X-Forwarded-For': '10.131.0.1'} gunicorn-web stdout | 2023-01-17 06:24:22,627 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,627 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,629 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:22,630 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:24:22,631 [232] [DEBUG] [app] Starting request: urn:request:c84a1798-7c0b-4ff2-bb14-01b01ddbded0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:22,631 [242] [DEBUG] [app] Starting request: urn:request:7d24d7c0-a0ce-4607-932d-1149b5cf1ff1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:22,631 [232] [DEBUG] [app] Ending request: urn:request:c84a1798-7c0b-4ff2-bb14-01b01ddbded0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c84a1798-7c0b-4ff2-bb14-01b01ddbded0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:24:22,632 [242] [DEBUG] [app] Ending request: urn:request:7d24d7c0-a0ce-4607-932d-1149b5cf1ff1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d24d7c0-a0ce-4607-932d-1149b5cf1ff1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '786d3e94', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:24:22,632 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:24:22,632 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:24:22,632 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:24:22,632 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:22,633 [239] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,634 [244] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,635 [239] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:22,636 [244] [DEBUG] [app] Starting request: urn:request:e46a3ed6-999e-4bae-a4a1-2a9c069948a1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:22,636 [244] [DEBUG] [app] Ending request: urn:request:e46a3ed6-999e-4bae-a4a1-2a9c069948a1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e46a3ed6-999e-4bae-a4a1-2a9c069948a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:22,636 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:24:22,637 [239] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:22,637 [244] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:22,637 [239] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:22,638 [241] [DEBUG] [app] Starting request: urn:request:16732967-8780-4cca-aeea-c1f08562d0c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:22,638 [241] [DEBUG] [app] Ending request: urn:request:16732967-8780-4cca-aeea-c1f08562d0c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16732967-8780-4cca-aeea-c1f08562d0c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:24:22,639 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:24:22,639 [244] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:22,639 [244] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:22,642 [239] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:22,644 [244] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:22,648 [239] [DEBUG] [app] Ending request: urn:request:cb84f918-e50a-4771-b10e-06175d187bc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb84f918-e50a-4771-b10e-06175d187bc1', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:22,648 [239] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.024 47 0.024) gunicorn-web stdout | 2023-01-17 06:24:22,648 [239] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:24:22,650 [244] [DEBUG] [app] Ending request: urn:request:6c7e7dad-f6e4-4284-b1cc-9e1afdb1d4a5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c7e7dad-f6e4-4284-b1cc-9e1afdb1d4a5', 'remote_addr': '10.131.0.1', 'http_method': 'GET', 'original_url': 'https://10.131.0.22/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'd17602e9', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:22,650 [244] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.025 47 0.026) gunicorn-web stdout | 2023-01-17 06:24:22,650 [244] [INFO] [gunicorn.access] 10.131.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityworker stdout | 2023-01-17 06:24:22,837 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:24:22,837 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:39.230038+00:00 (in 16.392252 seconds) securityworker stdout | 2023-01-17 06:24:22,838 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:52 UTC)" (scheduled at 2023-01-17 06:24:22.837365+00:00) securityworker stdout | 2023-01-17 06:24:22,838 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:24:22,838 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:24:22,841 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:24:22,843 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:24:22,849 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:24:22,851 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:24:22,851 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:52 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:24:24,642 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:24:24,642 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:48.830274+00:00 (in 24.188026 seconds) exportactionlogsworker stdout | 2023-01-17 06:24:24,642 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:24 UTC)" (scheduled at 2023-01-17 06:24:24.641816+00:00) exportactionlogsworker stdout | 2023-01-17 06:24:24,642 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:24:24,642 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:24 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:24:25,334 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:24:25,334 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:25:20.242198+00:00 (in 54.907587 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:24:25,334 [86] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:25 UTC)" (scheduled at 2023-01-17 06:24:25.334138+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:24:25,334 [86] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:24:25,335 [86] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:25 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:24:27,226 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:24:27,227 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:57.226424+00:00 (in 29.999508 seconds) buildlogsarchiver stdout | 2023-01-17 06:24:27,227 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:57 UTC)" (scheduled at 2023-01-17 06:24:27.226424+00:00) buildlogsarchiver stdout | 2023-01-17 06:24:27,227 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 24, 27, 227248), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:24:27,235 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:24:27,235 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:24:27,235 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:57 UTC)" executed successfully builder stdout | 2023-01-17 06:24:28,118 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:24:28,127 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:24:28,127 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:24:28,133 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:28,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:24:28,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:38.735066+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 06:24:28,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:38 UTC)" (scheduled at 2023-01-17 06:24:28.735066+00:00) notificationworker stdout | 2023-01-17 06:24:28,736 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:24:28,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 28, 736082), True, datetime.datetime(2023, 1, 17, 6, 24, 28, 736082), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:24:28,745 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:24:28,745 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:28,745 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:38 UTC)" executed successfully securityworker stdout | 2023-01-17 06:24:33,101 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-w2k29', 'process_name': 'securityworker.py', 'pid': '87'}